?

Log in

Gushi Systems' Journal
 
[Most Recent Entries] [Calendar View] [Friends]

Below are the 20 most recent journal entries recorded in Gushi Systems' LiveJournal:

[ << Previous 20 ]
Friday, June 15th, 2012
6:29 am
[gushi]
Server Moves!!!
Hey all,

I was forced to do a server move on fairly short notice tonight -- most sites should be up and running, with a few exceptions:

1) If you have a SQL database, I need to re-add these for you -- many people had databases on the order of multiple gigs of space, from long-abandoned phpBB installs. I have full backups and adding you back in only takes a second, just message me.

2) Webmail is not yet back. It's being worked on.

3) Other things I'm sure I'll find.

Send me a message at aim://GushiCell or just SMS me!
Thursday, August 12th, 2010
4:12 pm
[gushi]
Prime's taking a little nap, there's a minor power issue at the data center.

Mail should continue to flow in while things are down.

Apologies for the inconvenience.
Saturday, January 23rd, 2010
3:01 am
[xial]
Recent Downtime, and WordPress users.
Hi, everyone.

I'm Justin (some of you know me under this screen name), and I do some of the behind the scenes stuff here.
To bring those of you who aren't entirely aware of what's going on up to speed, I'll boil it down for you into a short synopsis:
  • On 18 Jan 2010, we suffered a hard drive failure. We had to get a drive put in to start the recovery process, which began that same afternoon.
  • The backup has most of what's there, but some files that were newer than the backup were, of course, missing. We (read: gushi ) have been reconstructing and replacing data as fast as possible.
  • There are always minor things that one must go through when getting things back into place. File ownership is one of those. I've been dealing with that as I am made aware of ownership issues.
To address one of the issues with files being newer than the backup, I spent 10 minutes writing up a quick and dirty script to deal with WordPress users. This one's for you:

If you are using WordPress, you may receive an instant message, tweet, or email from me, asking you to upgrade from a version of WordPress to the latest version.
There's usually a reason I am asking you to do it.
It's largely to protect you from some of the exploits seen in the wild for older, not patched holes in that platform.
It's also to help us out here, too -- cleaning up an exploited copy of WP isn't as high on my list at the moment: Getting all of our users up for the first time after our downtime is. Help me to help you, please.

I won't force-upgrade you to the latest version, unless YOU ask me to do it, or if your database tells me you are using a newer version than what the files say.

For the "What do you mean?" crowd, this is what I mean:
If I run my script, and see this:
WP Tattler 0.0.2 (OOH HE SAID A BAD WORD)
Local WordPress files claim to be: 2.6.3
But the database says: 2.6.3
Then I'll be messaging you to backup your database (there's plugins for that, of course) and upgrade.
 
 
If I run my script, and see this:
WP Tattler 0.0.2 (OOH HE SAID A BAD WORD)
Local WordPress files claim to be: 2.5.2
But the database says: 2.8.6
Then I'll drop a newer copy of WordPress in place for you, since you were trying to keep up to date.

I'm terrible with this public speaking thing, so this is where I shut up, and go back to what I'm used to.
Thanks for sticking with us through this.

- Justin

Current Mood: awake
Monday, January 18th, 2010
12:47 pm
[gushi]
Yes, I know prime is not answering.
There's a problem at the datacenter, I'm looking into it.

-Dan
Sunday, January 10th, 2010
10:17 am
[gushi]
Webmail Tweaks

Webmail's certificate is fixed.

I'm also in the process of moving webmail away from the shared apache space (which uses suPHP), and into its own apache processes, which will run as "webmail". Because there's no need to UID-switch, I can use the much faster mod_php5 for it.

It's presently up and running at webmailbeta.gushi.org (which uses a self-signed cert). At some point I may get a cacert.org cert for webmailbeta.
If you know how to override the cert issues, you can see it's quite a bit more speedy.

Tomorrow I'll probably move webmailproper over to it, and push webmailbeta up to the latest squirrelmail (1.4.19), .20 has been in "release candidate" mode for close to a year.

I also discovered the "attachments" directory was never getting purged. Whee, 700 megs of various attachments over the years.

Going to bed.

-Dan

Saturday, September 19th, 2009
4:47 am
[gushi]
If you run a forum, read this...

Hey there guys!

If you use a forum, of any sort, please for the love of god install a captcha.

If you're running phpBB, you can get a pretty good plugin for most software here. If you need help, just ask!

If you don't have FULLY captcha-fied registration, you're going to have your database overrun by spammers, who will see an open system and will continue to flood in, bloat your databases, and overload the system. My mail queue will fill up with mail destined to destinations that don't accept mail, and I'll get cranky. Nobody wants that.

If you're running IkonBoard, just delete it. It's dead, it's buggy, it's not being supported or actively checked, and it's not all that amazing.

Monday, April 30th, 2007
7:09 pm
[digoraccoon]
Support Question
While Gushi is recovering from his liver issues (God speed on recovery Gushi!) who would we see about issues on the prime server? I can't remember if there is a second in command who had access.

I noticed I couldn't access my emails all day today, I get an error "Preference file, /usr/local/squirrelmail/data/**username**.pref.tmp, could not be written. Contact your system administrator to resolve this issue".

Current Mood: curious
Monday, February 5th, 2007
6:55 pm
[gushi]
Did You Know?

That I use a ticket system. It has a direct email address which is: support@gushi.org

I am trying very hard to use it to track EVERYTHING time-related which I do, from simple support requests right on down to complex code changes, both for myself, as well as for others.

I am hoping that in this way I can track what my time is actually worth, even if I am not charging for it, i.e. to implement a (seemingly simple) feature on either my accounting engine, or the auction site I work with, or any of a dozen other projects I want to get off the grouns.

So if you have anything you need, it's best to email me there.

-Dan

Tuesday, January 9th, 2007
4:12 pm
[gushi]
Image Spam. I hate it. You hate it.

Heh heh. Evil ferret.

ifplugin Mail::SpamAssassin::Plugin::ImageInfo

body __GIF_200_500 eval:image_size_range('gif', 200, 200, 500, 500)
describe __GIF_200_500 Contains gif 200-500 (high) x 200-500 (wide)

meta GUSHI_SINGLE_MEDIUM_GIF ( __GIF_ATTACH_1 && __GIF_200_500 )
describe GUSHI_SINGLE_MEDIUM_GIF Message contains single inline gif 200-500x200-500
score GUSHI_SINGLE_MEDIUM_GIF 3.00

endif

That should deal with that.

For those who don't understand the above, I've added a new rule to SpamAssassin to increase the score of messages with a single gif of a certain size. Let me know if there's any stuff it catches that it should not catch, or vice versa.

Tuesday, December 19th, 2006
10:12 pm
[gushi]
Issues

Well, due to the expectation that power should be a constant, stable, normal thing to put into a computer being HEAVILY DISAPPOINTED several times today, prime lost a hard drive.

I had to reinstall the OS and restore -- no user data should be affected, please report any issues to myself or Xial.

Wednesday, December 6th, 2006
7:37 pm
[gushi]
Well, looks like a minor switching issue at the office. Prime is going to be out for a little until I can have it replaced. Should be another half hour or so from the time of this post.
Thursday, October 19th, 2006
12:27 pm
[gushi]
Mail Changes

I've just made some minor changes to the way mail is handled on prime.

Sendmail, the program that handles mail destined for the outside world, is designed to only queue mail (not deliver it) if the system load average gets too high. If load average is >8, it will queue the mail. If it gets higher than 12, it will outright refuse connections. These are designed to prevent a "downward spiral" effect, where more and more mail comes in and swamps the system. Mail queues in particular are very vulnerable when they get very large, but the system will "valve off" under ANY kind of load, not just mail server load.

This is the behavior some of you have been seeing. As some of the issues in my last post drove the processor mad, sendmail said "I'm going to sleep".

So, my new solution is as follows:

DaemonPurposeListens onQueue Directory
MTA (Mail Transport Agent)Transport of outside mail TO users.72.9.101.130, port 25/var/spool/mqueue
MSP (Mail Submission Program)Program called when sendmail is called from the command line, by some scriptsDoesn't listen, only sends mail to 127.0.0.1, with a daemon for periodic flush/var/spool/clientmqueue
new MSA (Mail Submission AgentTransport of users' OUTBOUND mail (i.e. from outlook, etc)localhost:25, *:587, *:2525, *:465/var/spool/msamqueue

The new agent is a full-blown mail server, so even in a case where it's one user mailing another, it will still run all the same checks and filters, still use the same certificates. However, since normal outside MAIL SERVERS will just requeue when the connection is refused, users tend to notice, get errors in outlook, and complain loudly. And since I know the load generated by users sending out mail (and the delivery of said messages) is FAR FAR lower than the 10-20 THOUSAND messages a day the normal MTA handles, so I have no problems with telling it NOT to shut down under load. And even if the MTA is down for some reason, it will still deliver outside mail directly.

What all does this mean to you?

As usual, when I make changes like this, it means test things and let me know if you spot any odd issues.

You should not have to change any settings at all.

Everything looks from my end like it works, but I use pine and submit all my mail locally. I can't possibly use all the MUA (Mail User Agents) that you guys do.

So let me know.

Wednesday, October 18th, 2006
4:45 pm
[gushi]
FUN WITH WEBSERVERS

Okay, some of you may have noticed some odd mail issues. It's been fun times, one thing after another. I feel like sharing.

First, there's a user whose UID is running IRC bots.

Their main site has a url like this: http://www.mysite.com/index.php?p=home

Now, where you have that p=home, you can put in any filename.php on the system in, so if I put in p=music, it would pull up music.php, and put it into the page. But people forget that any filehandle in php is ALSO usable as a url, so I could put in: http://www.mysite.com/index.php?p=http://www.gushi.org/gallery/index and it would bring up my gallery. What's worse, is any <?php> tags it encountered would be EXECUTED. So basically it's free license to run as much code as you want under this site.

This is why I hate PHP. "Rapid Deployment" my fucking ass. It allows people who DO NOT THINK to code up quick and dirty webpages WITHOUT CONTEMPLATING THE IMPLICATIONS.

So I shut down the rogue processes, added a few lines of php code that checked that the file didn't start with a dot, and actually exists in the target directory before running it.

Next, I find my mail queue is full of hundreds of outbound messages, all to AOL email addresses.

Apparently someone (a different user) had a homebrew mail script that did this:

//start building the mail string
$msg = "You Have Received an Information Request from Your Website";
$msg .= "Name: $_POST[name]";
$msg .= "Company: $_POST[company]";
$msg .= "E-Mail: $_POST[email]";
$msg .= "Message: $_POST[message]";

//set up the mail $recipient = "rpa@sp.com"; $subject = "Information Request"; $mailheaders = "MIME-Version: 1.0\r\n"; $mailheaders .= "Content-type: text/html; charset=ISO-8859-1\r\n"; $mailheaders .= "From: $_POST[email]"; //send the mail mail($recipient, $subject, $msg, $mailheaders);

Anyone else see the problem? How about the fact that you can feed the form a variable that makes your email address look like:

bob@bob.com[newline]BCC: joe@aol.com, steve@aol.com, john@aol.com....

So that got corrected too, with a simple regular expression:

if (pregmatch('\n.+$', $POST[email])) { echo "bad address"; die(); } -- meaning if the email variable has a newline before the very end, kill it.

THIRD (this just keeps getting funner)

There's a comic site: http://digitalunrestcomic.com/

Except their archive engine has a little flaw: when you try to view a nonexistent comic, like http://digitalunrestcomic.com/index.php?date=2005-10-30, the thing just hangs. It WAS hanging for the php max execution time, which I have set as high as 300 seconds.

The worst part? I try to go to the site for the php engine they're using: http://www.walrusphp.com/

Yeah, go view it.

This one got solved by dropping the user's php max execution time to ten seconds. Not a perfect solution, but it solves the issue nicely.

So that's it...things are once again running smoothly.

...and only two of you remembered sysadmin appreciation day.

Tuesday, September 19th, 2006
12:06 am
[xial]
Tell us if it's broken.
We're on the new box now.
Please do tell us if anything is broken.

As the users, you know what you need. We're here to help.

I'm sure we all have different usage patterns -- some of you use Perl, others PHP, and even a few use Python (which is fixed now, by the way).

Feel free to send me or Gushi a message (IM, Email) if there's something wrong.

- Xial

Current Mood: Still hungry...
Sunday, September 17th, 2006
3:06 am
[gushi]
Just a note
Prime's being migrated over to a new box, a little downtime of certain services is expected.

-Dan
Monday, July 10th, 2006
4:38 am
[gushi]
Upgrades
I've pushed prime.gushi.org up to the final version of FreeBSD 4. (4.11-STABLE)

Please report any oddness to me...I've streamlined a few things, and everything SHOULD be working as normal.

-Dan
Friday, June 30th, 2006
6:56 pm
[gushi]
Feeling Burnt
I'm tired, and a little mind-racing.

Basically, if you read my previous post, I need to move every mysql database on prime.gushi.org over somewhere else. Prime's running like a pig, and I think of all the things to move off, SQL will be the easiest. I'm moving it to quark for right now, but eventually it may get its own box. Users will be connecting to sql1.gushi.org, so we only have to go through this once.

Updates as I know them.

Current Mood: amused
Sunday, June 11th, 2006
10:25 am
[gushi]
Annoyance for the Evening
1) Notice mail is running slow.

2) Check mail queue, find 1000+ messages destined for brazil.

3) Figure out what user account (bassadmin for the site nicholaswalkerbass.com) is originating messages, thank god for suPHP, delete messages from queue.

4) run: tail -10000000 /usr/local/apache/logs|grep nicholas > /usr/local/apache/logs/nicholas.log

5) read above logfile for any POST requests:


nicholaswalkermusic.com 201.9.28.38 - - [11/Jun/2006:06:15:58 -0400] "POST /images/line.php HTTP/1.1" 200 0
"http://www.nicholaswalkermusic.com/images/line.php" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"


6) Deny ip from system -- ponder how to prevent this shit in future.

7) Go to www.dnsstuff.com to look up if this thing is in any abuse lists:
http://www.dnsstuff.com/tools/ip4r.ch?ip=201.9.28.38

8) Look at CBL (cbl.abuseat.org), think: this looks promising, has a low percentage of false positives, and I already use it for mail.

9) Download and install mod_access_rbl http://www.blars.org/mod_access_rbl.html, add it into apache server.

10) tail -F /usr/local/apache/logs | grep 403, look...



mdtas.com 125.247.126.10 - - [11/Jun/2006:08:25:32 -0400] "GET /addguest.html HTTP/1.0" 403 328 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"
kekeke.trichroic.net 218.232.162.130 - - [11/Jun/2006:08:25:39 -0400] "POST /mt/mt-comments.cgi HTTP/1.1" 403 344 "http://kekeke.trichroic.net/japan/archives/000035.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)"
sarah-davis.com 213.140.21.232 - - [11/Jun/2006:08:25:47 -0400] "GET /egg0501063li.gif HTTP/1.1" 403 341
"http://graficheaven.forumcommunity.net/?t=1656603&view=getlastpost" "Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"
sarah-davis.com 213.140.21.232 - - [11/Jun/2006:08:25:47 -0400] "GET /egg0501062li.gif HTTP/1.1" 403 341
"http://graficheaven.forumcommunity.net/?t=1656603&view=getlastpost" "Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"
kekeke.trichroic.net 218.149.163.158 - - [11/Jun/2006:08:25:50 -0400] "POST /mt/mt-comments.cgi HTTP/1.1" 403 344 "http://kekeke.trichroic.net/japan/archives/000035.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)"
magnetichiphop.com 125.234.64.216 - - [11/Jun/2006:08:25:56 -0400] "GET
/album/mastered/snippets/Magnetic%20North%20-%20Drift%20Away%20(album%20snippet).mp3 HTTP/1.1" 403 399 "-"
"NSPlayer/10.0.0.3802 WMFSDK/10.0"
arania.kamiki.net 213.192.60.36 - - [11/Jun/2006:08:25:59 -0400] "GET /sabintpacontest/Mecromace%20Sabin.jpg HTTP/1.1" 403 358 "http://www.hofyland.cz/main.php?kam=klub&order=zapis&idklubu=7840&1150028741" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
arania.kamiki.net 213.192.60.36 - - [11/Jun/2006:08:25:59 -0400] "GET /Gaia/anigif.gif HTTP/1.1" 403 338
"http://www.hofyland.cz/main.php?kam=klub&order=zapis&idklubu=7840&1150028741" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
kekeke.trichroic.net 61.41.133.99 - - [11/Jun/2006:08:26:02 -0400] "POST /mt/mt-comments.cgi HTTP/1.1" 403 344 "http://kekeke.trichroic.net/japan/archives/000035.html" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)"
catthouse.com 84.238.11.244 - - [11/Jun/2006:08:26:08 -0400] "GET /sitegraphics/bubble_rk_off.gif HTTP/1.1" 200 4030 "http://catthouse.com/kittens.html" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"



11) Think: Oh look, a lot of things that are suddenly being blocked are both IN the CBL (check here:
http://cbl.abuseat.org/lookup.cgi?ip=125.247.126.10&.submit=Lookup)...and are looking for commonly exploited things like guestbooks, movable type comment forms, etc...

12) Speak to #apache on freenode about possible ways to remove the dependency on foreign DNS, and also about easy ways to display useful error to the user as to why they're being blocked.

13) Be satisfied, and wander off to nap.
Tuesday, March 7th, 2006
6:29 pm
[gushi]
Anyone having issues with webmail...
We've been upgraded to SquirrelMail 1.4.6 -- if any of you (Jenna, Cubby, Saiyeeeeee) have been having trouble with it, give http://webmailbeta.gushi.org a shot.
Wednesday, March 1st, 2006
12:20 am
[gushi]
Light Ranting
Okay, let's say you have your primary email at AOL. Let's further say that you have a domain with me, and that you forward some or all of your mail to your AOL account.

Here's a VERY SIMPLE REQUEST.

If you can't be bothered to look at the headers, to tell which mail DID come through prime.gushi.org and which mail did NOT come through me, then either a) disable forwarding your mail to AOL, or b) stop using AOL's "report as spam" option entirely. You're not clued enough to be allowed to use it.

Here's why, kids.

Apparently, when AOL feels it receives too many "spam emails" from a system, they temporarily block it.

Now, if there's a virus out there, or a system just pumping spam, this is a good thing. It reduces load on AOL's mail servers.

Now, if your mail is being FORWARDED FROM ME -- AOL says "hey, prime.gushi.org is sending us spam!"

And, at one point, AOL was actually blocking us for a short period of time. This is the big problem. AOL thinks *I* am spamming you because people are spamming you here, it's being forwarded, and you report it.

If you want me to, I can make it so that mail originating from me has BIG OBVIOUS SIGNS like, [GS-DoNotMarkAsSpam] in the subject.
[ << Previous 20 ]
Gushi Systems   About LiveJournal.com