Project description
I need a multithreaded parser tag title.
Multithreaded parser of content tags title list URL from txt.file.
Sites on the list may or may not exist.
Implementing is desirable in PHP or some server language to run with VDS.
If it works well under the wind, it does not matter what.
Settings:
- incoming data file
- outgoing data file
- 100-1000 streams
- 3-10 seconds timeout
-
- file with a step (if suddenly it stops, so as not to parse from the beginning, but from this step)
The parser should not download all 100% of the page code, but only from the characters specified in the settings.
For example, from the beginning of the file, we skip 183 characters and write only 34 characters into the variable, which are parsim.
Something like $info = file_get_contents ($url, NULL, NULL, 138, 34).
There could be 50 million in total in the incoming base. Links.
Multithreaded parser of content tags title list URL from txt.file.
Sites on the list may or may not exist.
Implementing is desirable in PHP or some server language to run with VDS.
If it works well under the wind, it does not matter what.
Settings:
- incoming data file
- outgoing data file
- 100-1000 streams
- 3-10 seconds timeout
-
- file with a step (if suddenly it stops, so as not to parse from the beginning, but from this step)
The parser should not download all 100% of the page code, but only from the characters specified in the settings.
For example, from the beginning of the file, we skip 183 characters and write only 34 characters into the variable, which are parsim.
Something like $info = file_get_contents ($url, NULL, NULL, 138, 34).
There could be 50 million in total in the incoming base. Links.
Web development E-money
