Web Developer in need of Alch Prices and more...

[detoid9]

Hi to all, especially those you have knowledge in web development.

 

 

 

I am wanting to make some great online tools to aid RuneScapers in the area of High Alchemy and RuneScape items. I have made many Web 2.0 apps and other PHP / MySQL / Javascript mash-ups in my career, and have wanted to make some RuneScape tools for some time.

 

 

 

Unfortunately I haven't settled on a reliable source of High Alchemy prices that is easily accessible without having to do resource intensive operations like CURL.

 

 

 

So was just wondering if anyone out there has something that can fill this hole for me, and in return maybe I can give some coding assistance. I could even give the Tip.It Item Database a make-over if need be...

 

 

 

Some projects I had planned:

 

- A PHP on-demand GE price checker with caching. Everytime a user requests item information with prices older than so many hours, it will automatically update GE prices.

 

- High Alchemy adviser. Recommends best items to use with High Alchemy.

 

 

 

Languages I am very familiar with (code everyday in them): XHTML, CSS, PHP, Javascript. If Tip.It need any of my expertise, just send me a PM or something :)

[Lunar_Drifter]

Can't you just automate a PHP program to go through the Tip.It Item DB and grab the field for high alch? I guess it's just a number on the page and not a declared variable, but I'm sure there's a way to search for "high alch value: " that proceeds every item. Since each item is coded in by item ID, you could start your program at 0 and go all the way to X where X is the total number of entries. But you know that if you code every day. :ugeek:

[GlowinRedM]

Can't you just automate a PHP program to go through the Tip.It Item DB and grab the field for high alch? I guess it's just a number on the page and not a declared variable, but I'm sure there's a way to search for "high alch value: " that proceeds every item. Since each item is coded in by item ID, you could start your program at 0 and go all the way to X where X is the total number of entries. But you know that if you code every day. :ugeek:

 

 

 

This. Basically you would initially add all of the items that can be alched into a database after using a script to do it. You don't have to use cURL, just use file_get_contents("urlhere");. It will work just as well, and usually faster than cURL.

 

 

 

Then you would set a cron job to go through the GE database of all of the items (just use a for loop with file_get_contents through all of the item ids, but the first time make a blocklist of bad IDs with it) and get the price, and set the cron to run every 8-12 hours.

 

 

 

Then just display as a table :)

[detoid9]

Yeah hopefully I can find some hosting that doesn't have the annoying PHP execution timeouts. Crawling the entire GE database might take a few minutes and most cheap hosting solutions have around 30 second timeouts to save on resources. I'll probably be running this on my local server initially however.

 

 

 

I could probably get some free hosting off a friend on a dedicated server.

 

 

 

Lucky High Alch prices are static, so a one-time index should do fine. That should cut down a lot of work. Once the database is all initialised things should be all plain sailing. I may even make a Open Source PHP class for others to use ;)

 

 

 

A couple of concerns still remain though:

 

- Possible situation where Jagex fills in a bad ID with a new item? Not sure if that will ever happen however.

 

- How to know what is a bad ID and when to end the loop. E.g. How do I know I am at the last item? That is only if bad ID's are treated the same as going over the last ID number. Only way I am currently aware of determining this is if it encounters so many bad ID's in a row, it then declares it has reached the end of the item list.

 

 

 

Anyways, a bit of mind spillage. I may release a domain later down the track with some finalized idea's. If I do I will certainly be posting them back here.

[Lunar_Drifter]

Well the recently released Amulet of Ranging is item number 15,126 (item ID is listed in the URL). So probably 16,000 would be a good spot to stop now?

[detoid9]

Well the recently released Amulet of Ranging is item number 15,126 (item ID is listed in the URL). So probably 16,000 would be a good spot to stop now?

 

A completely automated system is what I'm going to aiming for, so setting a limit isn't really a option. There should be any gaps larger than 100 ID's, so I might set it at that.

 

 

 

Also another obstacle to hurdle over is pairing the alch prices with the GE prices. A straight text match might be somewhat un-reliable. But again I'll have to look into that a bit more.

[Lunar_Drifter]

At 15,126?

 

 

 

If you want it completely automated, you'll need to have it check for new items when RS is updated (you'll also need to find a way to check if RS is updated). I don't think there were any alchable items released since then. The GE DB only lists tradeable items, so things like the squad cards you get in MA aren't on the GE, but their item ID is still in effect past 15,126.

 

 

 

Also, the High Alch price isn't listed in the GE DB. It is in the Tip.it Item DB. The only thing is that the Tip.It Item ID is not equivalent to the GE DB (official) item ID. In fact, the Amulet of Ranging is item ID 8799 in the Tip.It Item DB as opposed to 15,126. So I don't know how you're going to automate that.

[detoid9]

Well I have the G.E. crawler at near completion. Now I have to incorporate the item stats and alch prices etc. I'm tossing up what site to use and how to link items together...

 

 

 

EDIT: G.E. crawler done :) It will take a hour or two to make the initial index though -.-

 

 

 

EDIT2: Now using multi-threading to speed the process up considerably. It's maxing out my internet connection :P On a 512kbs connection I am getting around 1000 pages every 20 minutes. Should go a lot faster on a server.

[detoid9]

Hmm the Tip.It Item Database is somewhat in complete with only 3000+ items in it. See the full list here: http://www.tip.it/runescape/index.php?r ... rch=Search

 

 

 

I might look else where for sourcing my Alch and Stats information... Any suggestions?

[Lunar_Drifter]

I don't think any item DB from a fan site will be complete to include all 15,000+ items.. I would suggest only automating the addition of alchable items, which you would have to enter the alch price by hand for each entry.. because- as far as I know- no item DB will have every single item. They will most likely contain every alchable item, but not in any way that is able to be systematically linked with the GE DB (other than matching name strings or some such).

[detoid9]

Well my crawler has finished indexing (just a few seconds ago), and with the initial index only 3163 of those 15000+ ID's are actually on the G.E. Database.

 

 

 

So I only need information for the 3100+ tradable items. So it shouldn't be too hard.

[detoid9]

OK guys I have finished my PHP crawler class. If you want a copy just PM me.

 

 

 

So far it will crawl the official G.E. Database, and a 'unofficial' item database to gather as much information as possible. Once the initial index is complete, a considerable amount of time is taken from following crawls, unless you tell it to do a complete re-crawl.

 

 

 

Enough of all this 'crawling', and time to go make some stuff with all this info!

[GlowinRedM]

This has made me want to make my own, haha, so I'm going to. When did you decide to tell the crawler to stop?

[detoid9]

This has made me want to make my own, haha, so I'm going to. When did you decide to tell the crawler to stop?

It works out when to stop, by setting a threshold that after so many bad ID's, it decides that it has reached the end.

[GlowinRedM]

alright thanks :) I did math and there are approximately 6 bad IDs for each good one, so setting a highish number for a bad ID (such as 50) would work well enough?