Softpedia >Windows >Internet >Search engine tools/submiting >Log File Analyser >  Changelog

Log File Analyser Changelog

What's new in Log File Analyser 6.0

Jan 9, 2024
  • New:
  • Updated Design & Unified Settings
  • Bytes Tab
  • Carbon Footprint Calculation
  • New Overview Charts
  • In-App Updates
  • Translated UI
  • Other Updates:
  • Version 6.0 also includes a number of smaller updates and bug fixes, outlined below:
  • There’s a new ‘Days Since Last Crawled Column’ which shows how long it’s been since a URL was crawled based upon the last response date, and last date of events available in the log file uploaded.
  • You’re now able to run multiple instances of the Log File Analyser, which provides the ability to compare projects side-by-side.
  • Zip64 files are now supported.
  • 7z and XZ compression are now supported.
  • The Windows installer has been updated with more options around saving a shortcut etc.
  • Version 6.0 has been updated to Java 21.

New in Log File Analyser 5.0 (May 23, 2022)

  • Updated Bot Verification:
  • Search bot verification for Googlebot and Bingbot has been updated to use their public IP lists, which were kindly provided by the search engines, rather than performing a reverse DNS lookup. This means you can verify search bots almost instantly – saving loads of time.
  • Other search bots, such as Baidu or Yandex still go through reverse DNS verification, so remove these from the default selection of a new project to speed up the process further, if they are not required.
  • Import Custom Log Formats:
  • The Log File Analyser has been upgraded to support a wider variety of log file formats automatically, and now provides the ability to view and customise fields to be used within the log file. While the LFA can automatically parse virtually all log formats, this is useful when log files are extremely customised, or a required field is missing.
  • You’re able to preview what’s in the log file and which log file components have been selected against which fields in the tool, and adjust if necessary. To enable this feature, select the ‘Show log fields configuration window on each import’ option under the ‘New Project’ configuration.
  • You can view what log file components are used for each field, customise or debug. This is an advanced feature, and in general is intended for more complex use cases.
  • Import CSVs & TSVs:
  • It can be a struggle to get hold of log files, and when you finally do – the log file data can be provided in random formats that are not raw access logs, such as CSVs or TSVs. Rather forcing you to try and get raw logs, you can now just upload these file types directly into the Log File Analyser.
  • Just drag and drop them into the interface in the same way you import log files and the LFA will automatically detect the log components, and upload the events for analysis.
  • Dark Mode:
  • If analysing log files wasn’t cool enough alone (and you hadn’t already realised from all the screenshots above), you can now switch to dark mode in the Log File Analyser.
  • To switch to dark mode, just hit ‘Config > User Interface > Theme’ in the top menu.
  • URL Events over Time:
  • The ‘URLs’ tab now has a lower window ‘Chart’ tab, which shows events over time in a graph when you click URLs.
  • This makes it easier to visualise crawling activity and trends of a page, than having to sift through the raw data in the ‘Events’ tab for each URL.
  • Exclude:
  • Similar to the already released Include configuration, you’re now able to provide a list of regexes for URLs to exclude from importing to further help focus on the areas you’re interested in analysing.
  • Countries Tab:
  • There’s a new ‘Countries’ tab, which shows data and a map of activity based upon the IPs from a log file.
  • This can be used in combination with the user-agent global filter to monitor whether a search engine is crawling from one specific location for example.
  • Apple Silicon & RPM for Fedora:
  • We’ve introduced a native Apple Silicon version, for those using shiny M1 macs and an RPM for Fedora Linux users.
  • In limited internal testing, we found that the native Silicon version was up to twice as fast importing log files than the emulation through Rosetta.

New in Log File Analyser 4.0 (Mar 25, 2020)

  • Wildcard User-Agent Matching:
  • You’re now able to use wildcard matching when configuring user-agents you wish to import into a project. This makes it far more flexible, particularly when user-agents strings change regularly, such as the new evergreen Googlebot and Bingbot.
  • You can choose from our pre-defined list of common search engine bots, or configure your own.
  • The default user-agent wildcard matching for Googlebot and Bingbot has also been updated to improve tracking of their ‘evergreen’ version naming.
  • Remove Parameters:
  • You’re now able to supply a list of parameters to strip from URLs and consolidate when you import log files.
  • This is available in the ‘new’ project configuration and is particularly useful when you have known parameters or issues, and need to consolidate data.
  • New JSON Timestamp Support:
  • In version 3.0, we provided support for log files in JSON format. There isn’t a common standard, so we have utilised customer provided JSON formats, and provided support for as many as possible.
  • This support has now been extended further to cover some less common JSON timestamp examples we have been provided by users. All you need to do is drag and drop in log files (or folders) as usual, and the Log File Analyser will automatically detect the format and analyze them.
  • Java 11 Update:
  • While this will make little practical difference to many users, behind the scenes we have updated to Java 11. Our SEO Spider will be following soon.

New in Log File Analyser 3.0 (Dec 11, 2018)

  • While this update is fairly small, it includes some significant new features and upgrades to the Log File Analyser based upon user requests and feedback. Let’s take a look at what’s new in version 3.0:
  • Configurable User-agents:
  • You’re now able to completely configure the user-agents you wish to import into a project. You can choose from our pre-defined list of common search engine bot user-agents, or de-select those that are not relevant to you.
  • This helps improve performance and reduces disk usage by focusing only on bots of interest. You can also add your own custom user-agents, which are then stored and can be selected for projects.
  • Previously the Log File Analyser only analysed bots for Google, Bing, Yandex and Baidu, so now this allows users to monitor bots from other popular search engines, such as Seznam in the Czech Republic. It also allows users to analyse and monitor other specific user-agents of interest, such as Google-News, or Adsbot etc.
  • Include Functionality:
  • Similar to the SEO Spider include feature, you’re able to supply a list of regular expressions for URLs to import into a project. So if you only wanted to analyse certain domains, or paths, like the /blog/, or /products/ pages on a huge site, then you can now do that to save time and resource – and more granular analysis.
  • New Log File Format Support:
  • The Log File Analyser now supports Application Load Balancing log file format and Verizon Edge Cast format.
  • All you need to do is drag and drop in log files (or folders) as usual, and the Log File Analyser will automatically detect their format and start to analyse them.
  • JSON Support:
  • You can now upload log files in JSON format to the Log File Analyser. There isn’t a common standard, so we have utilised customer provided JSON formats, and provided support for them all. We’ve found these to cover most cases, but do let us know if you have any issues when importing.
  • Other Updates:
  • We have also included some other smaller updates and bug fixes in version 3.0 of the Screaming Frog Log File Analyser, which include the following:
  • The overview graphs are now configurable, so you can easily select the date range and metrics to display.
  • The workspace storage location (where you store the database with log files) is now configurable.
  • X-Forwarded-For in W3C logs is now supported.
  • Time-taken in W3C logs is now supported.

New in Log File Analyser 2.0 (Mar 27, 2017)

  • Search Engine Bot Verification:
  • You can now automatically verify search engine bots, either when uploading a log file or retrospectively after you have uploaded log files to a project.
  • Search engine bots are often spoofed by other bots or crawlers, including our own SEO Spider software when emulating requests from specific search engine user-agents. Hence, when analysing logs, it’s important to know which events are genuine, and those that can be discounted.
  • The Log File Analyser will verify all major search engine bots according to their individual guidelines. For example, for Googlebot verification, the Log File Analyser will perform a reverse DNS lookup, verify the matching domain name and then run a forward DNS using the host command to verify it’s the same original requesting IP.
  • After validation, you can use the verification status filter, to view log events that are verified, spoofed or if there are any errors in verification.
  • Directories Tab With Aggregated Log Events:
  • There’s a new ‘directories’ tab, which aggregates log event data by path. If a site has an intuitive URL structure, it’s really easy to see exactly where the search engines are spending most of their time crawling, whether it’s /services/ pages, or the /blog/ etc. This view also aggregates errors and response codes ‘buckets’.
  • This aggregated view makes it very easy to identify sections of crawl waste or those that might need attention due to lack of crawl activity. The ‘directories’ tab can be viewed in a traditional ‘list’ format as well as the directory tree view.
  • Directory Tree View:
  • The directory tree view seen in the new ‘directories’ tab is now also available within other tabs, such as ‘URLs’ and ‘Response Codes’, too.
  • There’s a similar feature in our SEO Spider software, and it can often make visualising a site’s structure easier than traditional list view.
  • User-Agents, IPs & Referers Tabs:
  • You can now view aggregated log event data around each of these attributes, including number of events, unique URLs requested, bytes, response times, errors and more.
  • Analysing log event data aggregated around specific user-agents allows you to gain insights into crawl frequency, spread and errors encountered.
  • In an ‘all log event’ project, you can view every user-agent in a log file, including browsers, various other bots across the web, alongside search engine bots
  • The IPs tab allows you to view the most common IPs from which the search engine bots are making requests. Or perhaps you want to see specific IPs that have been spoofing search engines to crawl your site (before blocking them!).
  • Amazon Elastic Load Balancing Support:
  • We’ve had a lot of requests for this feature. The Log File Analyser now supports the specific AWS Elastic Load Balancing log file format.
  • This means you can simply drag and drop the raw access logs from Amazon Elastic Load Balancing servers straight into the Log File Analyser, and it will do the rest.
  • Other Updates:
  • We have also included some other smaller updates and bug fixes in version 2.0 of the Screaming Frog Log File Analyser 2.0, which include the following –
  • An ‘All Googlebots’ filter to make it easier to view all Googlebot crawlers at once
  • Users do not need to install Java for the application independently anymore
  • RAR file support
  • Bz2 file support

New in Log File Analyser 1.8 (Dec 26, 2016)

  • Support for W3C date-local and time-local fields
  • Prevent restoring to position off screen
  • Better detection of file types
  • Fix: Corrupt project when importing URL Data
  • Fix: Unable to import Apache log with response code but no response bytes field
  • Fix: Unable to parse IIS URI’s with spaces
  • Fix: Progress dialog appears over site url dialog preventing import
  • Fix: Crash on importing bad zip file

New in Log File Analyser 1.7 (Nov 8, 2016)

  • Fixed issue with graphs displaying without colour
  • Recently used projects list when no project is open
  • Minimise the number of fields required to import a log file
  • Support for Nginx request_time field
  • Support for W3C cs-uri and cs-host fields
  • Improved import performance
  • Improved performance when comparing with URL data
  • Handling of mid file format changes
  • Fixes for several crashes when faced with corrupt or invalid log files
  • Fixed issue with not being able URL Data files with data type changes

New in Log File Analyser 1.6 (Aug 17, 2016)

  • Fix for out of memory error when importing large Excel files
  • Linux version can now be run with openjdk and openjfx packages instead of oracles JRE
  • tar files can now be extracted, previously this had to be done outside of the LFA
  • Individual request line components for Apache log files now supported
  • IIS Advanced Logging Module W3C format now supported
  • W3C mid file format changes now supported
  • W3C x-host-header and cs-protocol now supported which allows multiple site urls to be read from W3C files if these are present

New in Log File Analyser 1.5 (Jul 30, 2016)

  • Fix for Windows users unable to launch with Java 8 update 101.
  • Reduced the number of mandatory fields required to import a log file.
  • Improved performance of ‘Not In Log File / Not In URL Data’ filters.
  • Fixed several crashes when importing invalid log files.

New in Log File Analyser 1.4 (May 18, 2016)

  • Fix for file in use by another process warning on Windows, that when importing the same file again incorrectly detected as a valid domain
  • User Agent detection fails to recognised Googlebot-Image/1.0

New in Log File Analyser 1.3 (May 4, 2016)

  • Support for Amazon CloudFront double percent encoded user agent fields
  • Fix for response code 499 causes log files not to be read
  • Fix for importing Apache logs with IPv6 hostnames
  • Fix for importing Apache logs with a duplicate field REQUEST_LINE
  • Support for logfiles in reverse chronological order
  • Fixed crash in progress dialog

New in Log File Analyser 1.2 (Apr 21, 2016)

  • Improved auto discovery of log file format on import.
  • Fixed crash when opening a project, on log file import, when mistakenly importing a trace file and canceling project creation on Windows.
  • Fix for drag and drop from WinRAR causing blank filenames.
  • Fix for issue with parsing of W3C log files that are delimited on tabs instead of spaces.

New in Log File Analyser 1.1 (Apr 13, 2016)

  • Fixed a bug when importing log files that require a site URL in non English locales.
  • Fixed a bug with auto discovering fields on import.
  • Fixed a bug where quitting during project creation led to corrupt projects.
  • The last open project is not reset if the app crashes.