Somewhere on the machine that hosts your website, a file is quietly recording every single visit. Every page load, every image, every bot, every failed login attempt. Most business owners never open it, which is a shame, because server logs are the closest thing your website has to a black box recorder, and they answer questions that analytics dashboards simply cannot.

We say this to clients all the time: Google Analytics tells you about the people who arrived; the logs tell you about everyone who knocked.

A log file is just a diary the server keeps automatically

Every time a browser (or a bot, or a scanner, or a search engine crawler) asks your server for something, the server notes it down. One request, one line. Millions of requests, millions of lines. There is no clever processing and no interpretation; it is a raw, chronological record of what was asked for and what the server said in reply.

Two files usually matter. The access log records successful and unsuccessful requests. The error log records the things that went wrong in a way the server thought worth complaining about, which is where you look when a page is white, broken or stubbornly refusing to load.

What a line in an access log actually contains

A typical entry follows a long-standing format, and once you know the pieces, it stops looking like noise.

• IP address: where the request came from. Useful for spotting one source hammering your site repeatedly.
• Timestamp: the exact moment of the request, which is how you line up a spike in traffic with the moment your site slowed down.
• Request line: the method (usually GET or POST) and the path requested, so you can see precisely which page or file was asked for.
• Status code: the server’s verdict. 200 means fine, 301 and 302 are redirects, 404 means not found, 403 means forbidden, and anything starting with 5 means the server itself had a problem.
• Bytes sent: how much data went out, which is how you discover that one enormous uncompressed image is eating your bandwidth.
• Referrer: the page that linked to the request, where the browser chose to share it.
• User agent: a string identifying the browser or bot. This is how you tell Googlebot apart from a real customer, and both apart from a scraper pretending to be Chrome.

Why server logs are worth an hour of your attention

Analytics tools rely on JavaScript running in a visitor’s browser. That means they miss anything that never gets that far: blocked requests, bots, crawlers, failed page loads, and visitors with tracking blockers. Logs miss nothing, because the server cannot serve a request without recording it.

• Diagnosing outages: the error log usually names the exact file and line that failed, which turns a mystery into a job.
• Understanding crawl behaviour: you can see exactly which pages search engines crawl, how often, and where they hit dead ends.
• Finding broken links: a pile of 404s tells you which URLs people and search engines still expect to exist.
• Spotting attacks: repeated failed logins from one address, or thousands of requests for files you do not have, is the fingerprint of an automated attack.
• Explaining slow pages: response sizes and timings show you what is heavy long before a customer complains.
• Settling arguments: when someone insists the site was down at 3pm, the log either agrees or it does not.

How to read your logs without becoming a systems administrator

Step 1: Find out where they live

On most hosting, logs are available in the control panel under a heading such as Raw Access Logs, Logs or Metrics. On a VPS they are typically in a directory like /var/log, and you will reach them over SSH.

Step 2: Look at the shape before the detail

Do not start reading line by line; you will lose an afternoon and learn nothing. Start with counts. How many 404s? How many 500s? Which pages appear most often? Which IP addresses appear most often?

Step 3: Filter to the moment that matters

If the site broke on Tuesday at 2pm, look at Tuesday at 2pm. The error log entry nearest the failure is usually the one that explains it.

Step 4: Separate humans from machines

Group by user agent. A surprising proportion of traffic to a small business website is not human, and mistaking bot traffic for a marketing success is an easy way to draw the wrong conclusion.

Step 5: Turn findings into fixes

A cluster of 404s becomes a redirect map. A slow, heavy page becomes an image optimisation job. A hammering IP address becomes a firewall rule. Logs are only valuable when they end in a decision.

Logs, analytics and monitoring do different jobs

• Server logs: complete, raw, unfiltered, and blind to nothing. Excellent for technical truth, poor for reading trends at a glance.
• Analytics tools: friendly dashboards, good at behaviour and conversions, but they only see visitors whose browsers run the tracking script.
• Uptime monitoring: tells you the site is down, usually before a customer tells you, but it will not tell you why.
• Application logs: generated by WordPress or your plugins, and often the fastest way to identify a misbehaving plugin.
• Search Console: shows how Google sees your site, and pairs beautifully with logs when you are chasing crawl problems.

Use them together. Any one of them alone will mislead you eventually.

Best practices for handling logs sensibly

• Know your retention: many hosts rotate logs every few days, so the evidence you need may already be gone. Check before you need it.
• Archive what matters: if you are investigating something over time, copy the logs somewhere safe.
• Watch disk space: a runaway error log can fill a server and take the site down all by itself, which is a particularly annoying way to have an outage.
• Treat them as personal data: IP addresses can be personal data under UK GDPR, so do not keep logs indefinitely and do not share them casually.
• Automate the boring part: a simple weekly summary of 404s and 500s is worth more than an occasional heroic deep dive.
• Do not log secrets: passwords and tokens should never appear in a log file, and if they do, that is a bug worth fixing today.

Common mistakes we see all the time

• Only looking after a disaster: logs are most useful when you know what normal looks like.
• Panicking at 404s: some are inevitable. It is the pattern, not the presence, that matters.
• Trusting the user agent: anything can claim to be Googlebot. Verify before you draw conclusions.
• Blocking too aggressively: an over-enthusiastic firewall rule can lock out search engines, or your own customers.
• Ignoring the error log entirely: it is often shouting the answer while everyone guesses.
• Keeping everything forever: unlimited retention is a data protection liability, not a badge of diligence.

Where log analysis is heading

The trend is towards logs being read by software rather than by people. Hosting platforms increasingly ship with dashboards that surface anomalies automatically, and SEO tools now offer log-file analysis that shows crawl budget being wasted on pages nobody cares about. That is a genuinely useful direction, because the value was never in reading the lines; it was in noticing the patterns.

At the same time, privacy expectations are tightening, and thoughtful teams are anonymising IP addresses and shortening retention windows. Expect the future to involve fewer raw text files on servers and more structured, searchable, shorter-lived records. Which is fine, as long as somebody is still looking.

Your server log checklist

• Locate your access and error logs and note how long they are kept.
• Count your status codes and look for anything in the 400s and 500s.
• Build a redirect map from the most common 404s.
• Identify your heaviest pages and optimise the obvious offenders.
• Separate bot traffic before you judge your marketing.
• Check disk usage so logs cannot fill the server.
• Agree a retention period and stick to it.

A worked example: the mystery of the slow Tuesday

Here is the sort of thing logs are for. A client tells you the website felt sluggish on Tuesday afternoon, then fine again by the evening. Analytics shows nothing unusual, because most of the traffic involved never ran a line of JavaScript.

Open the access log for Tuesday. Count requests per hour, and there it is: a tenfold spike between two and four. Group those requests by IP address and a handful of addresses account for nearly all of it. Look at what they asked for and it is not your product pages at all; it is a long, mechanical march through every combination of filter and sort parameter on your shop, or repeated probing at a login path that does not exist.

That is not a marketing win. That is a badly behaved crawler or a scanner, and the fix is a rate limit, a robots directive, or a firewall rule, not a bigger server. Without the log you would have spent the week wondering whether to upgrade your hosting; with it, the answer took twenty minutes.

Turning 404s into recovered traffic

The single most valuable thing most small businesses find in their logs is a list of pages people still want that no longer exist. Old blog posts, product pages from a previous website, links printed in a leaflet three years ago.

Every one of those 404s is somebody, or some search engine, arriving with intent and being handed a dead end. Map the common ones to the closest relevant page with a permanent redirect and you recover traffic you have already paid for, and rankings you had otherwise abandoned. It is not glamorous work; it is just quietly, reliably profitable.

Where do I find the log files for my website?

Start in your hosting control panel; most providers expose raw access logs and error logs there. On a VPS or dedicated server they live on disk, usually under /var/log, and you will need SSH access to read them. If you cannot find them at all, your host will tell you in one email.

What is the difference between an access log and an error log?

The access log records every request, successful or not. The error log records problems the server encountered, such as a script failing or a permission being denied. When something is broken, start with the error log; when you want to understand traffic, start with the access log.

Do server logs replace Google Analytics?

No, and they are not trying to. Analytics is far better at understanding what humans did once they arrived. Logs are far better at understanding what actually reached the server, including everything analytics never sees. Serious diagnosis usually needs both.

Can log files slow my website down?

Writing them costs almost nothing. Letting them grow unchecked can absolutely cause problems, because a full disk will stop a site dead. Log rotation exists precisely to prevent this, and it should be switched on.

Are server logs covered by data protection rules?

They can be, because an IP address may count as personal data. That does not mean you cannot keep logs; it means you should have a sensible retention period, a reason for keeping them, and reasonable security around them. If in doubt, keep less for less time.

Let us look under the bonnet for you

Most websites are throwing off useful signals every hour of every day, and almost nobody is listening. If your site is slow, occasionally broken, or quietly haemorrhaging traffic to pages that no longer exist, your server logs almost certainly know why. We can dig through them, fix what they reveal, and build you a website and hosting setup that stops generating nasty surprises in the first place. Contact Us and let us take a proper look.