rannmann

Keep in mind it's a HTTP 200 status response, alongside Content-Length being 0 - just thought i'd clarify that! They should return a 204 then! I have to say, their throttling is pretty intense. It makes me wonder if they're using IIS or something equally as terrible and can't handle many reqs/sec. If any Jagex employee reads this, you should try out nginx. A web server with nginx can easily handle 800,000 HTTP req/sec, and the resulting CPU load is minimal (usually around 1% total). It took 4 hours 41 minutes and 19 seconds to load just 3472 pages, and had to sleep 1228 times, averaging 2.827 requests per throttle and 1 page every 4.86 seconds. Once again, if anyone wants this data, it's available here: https://docs.google.com/spreadsheet/ccc?key=0AoQ7nO6Bv5TvdHVLbmgyWTdxck9lOERabHdkbl92enc The format is: ID, Item Name, Item Description, Price on March 2nd 2012 (I don't plan on keeping it up to date)

I don't see a way to grab item descriptions by searching multiple items at a time. I can still use the API to minimize bandwidth, but that still means I'll have to crawl each item individually. However, since I have list of all the items already, I think I'll just read the pages I know exist and cut down on 20k requests. In regards to the 200s, I'll just throw in a check for that, and wait some number of seconds before trying again. Thanks for the help :)

Also, I'm not sure if my web crawler is working correctly as I only have about 883 items total from the GE site (not using the API), and I've searched 1-23609 thus far. Does anyone know how many items can be sold on the GE? All the blanks spots here were things that failed to be crawled correctly (or are not able to be sold/bought on the GE): https://docs.google.com/spreadsheet/ccc?key=0AoQ7nO6Bv5TvdHVLbmgyWTdxck9lOERabHdkbl92enc#gid=0

For anyone on Linux, you can sort the list given above by ID with the following: sort -n -t\; +1 -2 itemList.txt should come out something like this: Spellstorm drape;23052 Raw wolf meat;23058 Poorly-cooked bird meat;23060 Poorly-cooked beast meat;23062 Rabbit sandwich;23065 Item list sorted by Name: https://docs.google.com/spreadsheet/ccc?key=0AoQ7nO6Bv5TvdFVqekJ6WkpHalpqV3E3cE9Hd1diVmc Item list sorted by ID: https://docs.google.com/spreadsheet/ccc?key=0AoQ7nO6Bv5TvdDJ4clAzNnpJUWZZeVpsREN4VndnYlE

I saw that, but getting .net to work on Linux has always been a royal PITA. I'll just let my crawler do its thing. I decided I want to grab some more data about the items while I'm at it, such as item descriptions. I see his is actually a little smarter than mine, as mine has to load over 22k pages (I'm not really up on all the RS stuff, honestly). I wasn't aware of this item database query directly, and was just scanning every item through the grand exchange, whether or not it existed on their page. If I ever need to do this again, I'll keep this in mind. Thanks for the reply. :) Edit: Hoozah! Here is the data for everyone's enjoyment: https://docs.google.com/spreadsheet/ccc?key=0AoQ7nO6Bv5TvdHVLbmgyWTdxck9lOERabHdkbl92enc

Finished the crawler. I did a quick test on 20 items, and it seems to dish out 1.166 requests per second, meaning items 0-23000 (I think that's a little past the upper limit) will take around 5h28m. I don't want to do this to my poor webhost, so I'll boot up my server in the morning and let it run then. Hopefully I'll be able to get more req/sec on my box, but we'll see.

I doubt anyone is up this late, but I figured out of all the sites, this might be the best bet. I was taking a look at item codes, because I need to have a database of Names->IDs for optimal performance with this project I'm working on (only need things that can be sold on the GE), and I noticed tip.it uses different item codes than the item database on the official GE site. Does anyone know why this is? I haven't been able to find a database anywhere on the web, so I was just going to quickly throw together a crawler tonight to make a CSV list for me so I can import it into my MySQL database (I want to check them first to make sure nothing funky is going on). In all honesty, I'll probably be done by the time anyone responds to this, but if you end up teaching me something, or saving me any amount of time, it would be appreciated! Thanks in advance, -rann

I have class when your party starts. :(

I HART Zez (although I've only talked to him for like 5 minutes on IRC), the old nite (only what I heard from his friends), lwc, METALLICA, and oddfaery! YOU ALL ROCK MY WORLD!!! Once again, HART! (doesn't mis-spelling make me LEET?!)

Wow you're dense. Okay, YOU'RE NOT THE GUY WHO STOLE MY RED PHAT. I never said you were, and you weren't the friend I was talking about. I barely knew you on vent... and it might not even have been you. It coulda' been someone you were loaning your account to. In any case, SOMEONE "borrowed" my d-chain, and gave me TosilX. He gave me fake recovs, stole the account back, and kept the d-chain. I was logged on to TosilX, so if it wasn't you that gave me the account, then it was one of your friends who trains for you. Like I said before, everyone who reads all the posts and does a minute bit of research will come to the same conclusion everyone I know has. You are a scammer. I don't care the reason you got banned, you deserved it. You should have been banned a long time ago, and I'm glad Jagex finally found a reason to do so. Have a lovely day.

Who are you in chat? And if you didn't know my ban was sorted out with Shirl and a few other admins. There was a 'situation'. and rannmann, I don't know you lol. First you say I stole your dragon chain and now I've stolen your phat also? Stop changing your story lol Okay, well apparently you didn't read it very well. I never said YOU stole my red phat. I said someone else did because I trusted him. I used this example rather than the dchain example because I didn't really trust you too much with it. I had a feeling you might steal it, but at the time I was rich enough so it wouldn't make too much of a difference. And apparently you didn't settle your ban with shirl, because she has the power to override every ban. [06] * Tosi ([email protected]) has joined #runescape [06] * ChanServ sets mode: +b *!*@Tosi.youre-currently-packeting-localhost.gamesurge [06] * Tosi was kicked by ChanServ ((Unbrokenxero) Tosi, known scammer and cheater, enough said.) and then the evade attempt [06] * Rapine ([email protected]) has joined #runescape [06] <@Xero> !k Rapine You were banned for a reason, now stay out, just cuz you can use java chat doesn't make it right. [06] * Rapine was kicked by ChanServ ((Xero) You were banned for a reason, now stay out, just cuz you can use java chat doesn't make it right.) I think now almost everyone can poke a hole in your story. If you're gonna' lie, lie well.

Aww.. 10:00 am alaska time... :( I'm not getting up at 10:00 on a saturday. How long do you think this will go on?

lol odd, he's just gonna' say "Who are you? What's #runescape? Gamesurge? Never heard of it... Stop being rude."

If you guys didn't ready my little shindig, or did and don't beleive it, I could care less. I'm just warning everyone that you shouldn't trust internet people. You can post "I pity you" or whatever, just don't give him anything you earned. The best scammers are the ones you don't expect. I knew this one guy for over a year, and then the minute I let him borrow a red phat, he logs and I never see him again. 1 year of friendship for some pixel money... people are willing to use other people's trust against them. Be careful everyone.

I usually don't post... or even read these forums, but one of my friends said something about TosilX, and this thread being hillarious... Of course I've had my run-ins with TosilX... I don't play rs anymore, so I'll come out with it... ALL of it. Near the end of my RS "career" I suppose you could call it, I began heading down the black market path. I came across a lot of rs money (at the time about 115M, now it'd be around 330M), and was planning on selling the mils. If any of you wanna try this, I just have to say DON'T TRY IT! It's not worth it. I sold maybe 8M and I got scammed out of a red phat, d-chain, and a few mils. Not worth it. But here's where this part comes in. A person I met on ventrilo was willing to give me one of his accounts (he had at least 4), if I would let him borrow a Dragon Chain for a few days until he sold one of his accounts. Well... I let him. He gave me the TosilX and recoveries... but guess what!? The recovs didn't work, and he stole the account back along with the d-chain. Hrm... go figure. At the time I was still rich and had two d-chains, so it wasn't too big of deal. The reason I did it was to try rune mining. I don't beleive this story one bit... He was an [wagon] on vent, scammed me out of 28M, and actually gave me this account at one point. You'd think Jagex would notice the IP changing from one place in the US to Alaska in about 10 seconds. But TosilX SHOULD be banned for account sharing, account trading, and real life item trading. If I remember correctly he also keylogged people... but that part I cannot confirm. There were a lot of people on that vent who keylogged and I have a hard time keeping the straight after over half a year of not talking to any of them... Anyone remember the swift-switch.net keylogging scam? Those were some of thet people from that vent whom TosilX associated with, and called his "friends". Don't pity this kid, and moreover, don't give him any of your hard-earned money/items.