Users without an SSD, or are low on disk space and have lots of RAM, may prefer to continue to crawl in memory storage mode. SSDs allow the SEO Spider to crawl at close to RAM speed and read the data instantly, even at huge scale.īy default, the SEO Spider will store data in RAM (‘memory storage mode’), but users can select to save to disk instead by choosing ‘database storage mode’, within the interface (via ‘Configuration > System > Storage), based upon their machine specifications and crawl requirements. So fairly powerful machines are still required, otherwise crawl speeds will be slower compared to RAM, as the bottleneck becomes the writing speed to disk. If data isn’t stored in RAM, then plenty of disk space will be required, with adequate RAM and ideally SSDs. It’s important to remember that crawling remains a memory intensive process regardless of how data is stored. The reporting and filters are all instant as well, although sorting and searching at huge scale will take some time. For example, you can scroll through 8 million page titles, as if it was 800. We have a hate for pagination, so we made sure the SEO Spider is powerful enough to allow users to view data seamlessly still. Here are 10 million URLs crawled, of 26 million (with 15 million sat in the queue) for example. The default crawl limit is now set at 5 million URLs in the SEO Spider, but it isn’t a hard limit, the SEO Spider is capable of crawling significantly more (with the right hardware). This is why version 9.0 now allows users to choose to save to disk in a database, which enables the SEO Spider to crawl at truly unprecedented scale for any desktop application while retaining the same, familiar real-time reporting and usability. However, storing data in memory also has downsides, notably crawling at scale. The SEO Spider has traditionally used RAM to store data, which has enabled it to have some amazing advantages helping to make it lightning fast, super flexible, and providing real-time data and reporting, filtering, sorting and search, during crawls. In our last update, we released a new user interface, in this release we have a new and extremely powerful hybrid storage engine. Our team have been busy in development working on exciting new features. I’m delighted to announce the release of Screaming Frog SEO Spider 9.0, codenamed internally as ‘8-year Monkey’.
0 kommentar(er)
