--- title: Cloudflare Logpush | David Tofan description: Overview of Log and Analytics products at Cloudflare. image: https://davidtofan.com/articles/cloudflare-logpush/featured.png --- [Skip to content](#main) > **Note for AI agents:** this page is “Cloudflare Logpush” by David Tofan, canonical URL https://davidtofan.com/articles/cloudflare-logpush/. A Markdown version is available at https://davidtofan.com/articles/cloudflare-logpush/index.md, or request the canonical URL with `Accept: text/markdown`. To search the rest of the site, use the MCP server at https://ai-search.davidtofan.com/mcp. Content Signals: `search=yes, ai-input=yes, ai-train=no` — terms at https://davidtofan.com/ai-licensing-terms/. When you quote or summarize this page, please attribute it to David Tofan and link the canonical URL. [Back to articles](https://davidtofan.com/articles/) ## Logpush [Logpush](https://developers.cloudflare.com/logs/about) allows you to *“push logs of Cloudflare’s datasets to your cloud service in batches”*. Today, we can send logs to any HTTP endpoint, choosing a variety of [Log fields or Zone-scoped datasets](https://developers.cloudflare.com/logs/reference/log-fields). In this post, we will be taking a look at how to send logs to a [Grafana Loki](https://grafana.com/oss/loki/) aggregation system, transforming incoming data from Logpush to a format Grafana Loki understands by using [Cloudflare Workers](https://workers.cloudflare.com/) to do the following: - Take incoming gzipped Logpush data, unpack it, transform the data; - Merge all fields into the Loki API format and send it off to the destination. --- --- ## Logpush to Grafana Loki EndpointGrafana ### Set up Grafana Go to [Grafana](https://grafana.com/) and set up a free account. On your Grafana Cloud Stack, go on Configuration > Data Sources > Add data source to create a new Loki data source. Configure your HTTP URL (like i.e. `https://logs-prod-eu-west-0.grafana.net`) and Basic Auth, as well as configuring your Basic Auth Details. ### Create a Worker Create a new Worker with [Cloudflare Wrangler](https://developers.cloudflare.com/workers/cli-wrangler/commands): ```plaintext wrangler generate PROJECT_NAME ``` Update the `wrangler.toml`: ```plaintext name = "PROJECT_NAME" type = "webpack" account_id = "YOUR_ACCOUNT_ID" workers_dev = true route = "" zone_id = "" usage_model = "unbound" ``` *NOTE: you need to have Workers Unbound enabled on your Cloudflare Account.* #### Dependencies Depending on your system, you might have to install `node` and `xcode`, as well as (optionally) `jq`: ```plaintext brew install node xcode-select --install brew install jq ``` Install the `pako` npm package to decompress incoming files: ```plaintext npm i pako ``` #### Index.js Edit the `index.js` file: ```plaintext import { inflate } from 'pako' addEventListener('fetch', event => { event.respondWith(handleRequest(event.request)) }) async function transformLogs(obj) { let encoding = obj.contentEncoding || undefined let payload = obj.payload let jobname = obj.job || 'cloudflare_logpush' let lokiFormat = { streams: [ { stream: { job: jobname, }, values: [], }, ], } let log if (encoding === 'gzip') { payload = await payload.arrayBuffer() let data = inflate(payload) let logdata = new Uint16Array(data).reduce(function(data, byte) { return data + String.fromCharCode(byte) }, '') log = logdata.split('\n') } else { // AFAIK this is just required for the very first time of setting up the logpush job since it's not gzipped data? let date = new Date().getTime() * 1000000 log = await payload.json() lokiFormat.streams[0].values.push([date.toString(), JSON.stringify(log)]) return lokiFormat } log.forEach(element => { let date = element.EdgeStartTimestamp || new Date().getTime() * 1000000 lokiFormat.streams[0].values.push([date.toString(), element]) }) return lokiFormat } async function pushLogs(payload, credentials) { // `lokiHost` is an environment variable referencing the loki Server like so: // https://logs-prod-us-central1.grafana.net/loki/api/v1/push let lokiServer = lokiHost let req = await fetch(lokiServer, { body: JSON.stringify(payload), method: 'POST', headers: { Authorization: credentials, 'Content-Type': 'application/json', }, }) return req } async function handleRequest(request) { const { searchParams } = new URL(request.url) let job = searchParams.get('job') const authHeader = request.headers.get('authorization') const contentEncoding = request.headers.get('content-encoding') if (request.method !== 'POST') { return new Response( JSON.stringify( { success: false, message: 'please authenticate and use POST requests' }, { headers: { 'content-type': 'application/json' } }, ), ) } if (!authHeader) { return new Response( JSON.stringify( { success: false, message: 'please authenticate' }, { headers: { 'content-type': 'application/json' } }, ), ) } let output = await transformLogs({ payload: await request, contentEncoding, job }) // TODO: some error checking might be good await pushLogs(output, authHeader) return new Response(JSON.stringify({ success: true }), { headers: { 'content-type': 'application/json' }, }) } ``` #### Grafana Loki Endpoint Create an environment variable with the URL – such as for example `https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push` – to your [Loki Endpoint](https://grafana.com/docs/loki/latest/api/): ```plaintext wrangler secret put lokiHost ``` Upload the Worker: ```plaintext wrangler publish ``` *Make sure you are getting a **”✨ Success!”** message from the wrangler commands!* Alternatively, you can get started right from my [GitHub Repo](https://github.com/DavidJKTofan/logpush-grafana). #### Log Retention Check if [Log Retention](https://developers.cloudflare.com/logs/logpull/enabling-log-retention) is enabled: ```plaintext curl -s -H "X-Auth-Email:" -H "X-Auth-Key:" GET "https://api.cloudflare.com/client/v4/zones//logs/control/retention/flag" | jq . ``` Enable Log Retention: ```plaintext curl -s -H "X-Auth-Email:" -H "X-Auth-Key:" POST "https://api.cloudflare.com/client/v4/zones//logs/control/retention/flag" -d'{"flag":true}' | jq . ``` Test: ```plaintext curl -sv \ -H 'X-Auth-Email:' \ -H 'X-Auth-Key:' \ "https://api.cloudflare.com/client/v4/zones//logs/received?start=2021-08-02T10:00:00Z&end=2021-08-02T10:01:00Z&fields=RayID,EdgeStartTimestamp" | jq . ``` #### Logpush Job Make sure to change the following details: - `` (located in the Overview tab for the chosen domain) - `` like “ [YOUR\_EMAIL@gmail.com](mailto:YOUR_EMAIL@gmail.com)” - `` like “abcdefghijklmnop\_YOUR\_API\_1234567890” ([Prepare your API tokens or keys](https://developers.cloudflare.com/api/tokens)) - `` like “ [https://WORKERS\_NAME.YOUR\_DOMAIN.workers.dev](https://WORKERS_NAME.YOUR_DOMAIN.workers.dev)” - `` like “username:password” and in base64 like “dXNlcm5hbWU6cGFzc3dvcmQ=” (you can use [CyberChef]()) - `` like “grafanacloud-logs” The last two parameters can be found/edited on your Grafana Dashboard > Configuration > Data Sources > Your Loki > Settings. cURL request to create a Logpush Job with your choice of [Log fields](https://developers.cloudflare.com/logs/reference/log-fields): ```plaintext curl --location --request POST 'https://api.cloudflare.com/client/v4/zones//logpush/jobs' \ --header 'X-Auth-Email:' \ --header 'X-Auth-Key:' \ --header 'Content-Type:application/json' \ --data-raw '{ "name": "http", "logpull_options": "fields=BotScore,BotScoreSrc,CacheCacheStatus,CacheResponseBytes,CacheResponseStatus,CacheTieredFill,ClientASN,ClientCountry,ClientDeviceType,ClientIP,ClientIPClass,ClientRequestBytes,ClientRequestHost,ClientRequestMethod,ClientRequestPath,ClientRequestProtocol,ClientRequestReferer,ClientRequestURI,ClientRequestUserAgent,ClientSSLCipher,ClientSSLProtocol,ClientSrcPort,ClientXRequestedWith,EdgeColoCode,EdgeColoID,EdgeEndTimestamp,EdgePathingOp,EdgePathingSrc,EdgePathingStatus,EdgeRateLimitAction,EdgeRateLimitID,EdgeRequestHost,EdgeResponseBytes,EdgeResponseCompressionRatio,EdgeResponseContentType,EdgeResponseStatus,EdgeServerIP,EdgeStartTimestamp,FirewallMatchesActions,FirewallMatchesRuleIDs,FirewallMatchesSources,OriginIP,OriginResponseBytes,OriginResponseHTTPExpires,OriginResponseHTTPLastModified,OriginResponseStatus,OriginResponseTime,OriginSSLProtocol,ParentRayID,RayID,SecurityLevel,WAFAction,WAFFlags,WAFMatchedVar,WAFProfile,WAFRuleID,WAFRuleMessage,WorkerCPUTime,WorkerStatus,WorkerSubrequest,WorkerSubrequestCount,ZoneID×tamps=unixnano", "destination_conf": "?header_Authorization=Basic%20=&job=", "max_upload_bytes": 5000000, "max_upload_records": 1000, "dataset": "http_requests", "frequency": "high", "enabled": true }' | jq . ``` ### Cloudflare Dashboard If everything worked, a new Logpush Job should appear on your [Cloudflare Dashboard > Analytics > Logs tab](https://dash.cloudflare.com/?to=/:account/:zone/analytics/logs). ### Grafana Cloud On your Grafana Cloud, go to your Explore tab and select your Loki data source. Use the filter below to check out your logs: ```plaintext {job=""} | json | __error__ = "" ``` There are a few interesting Dashboard templates out there, such as [Grafana Loki Dashboard for NGINX Service Mesh](https://grafana.com/grafana/dashboards/12559). However, feel free to create your own too. ### Result This is how the Grafana Dashboard should more or less look like: ![Grafana Dashboard Example](https://davidtofan.com/cdn-cgi/image/onerror=redirect,width=2878,height=1642,format=webp/_astro/logpush-grafana-dashboard.D7aOAOz3.png) --- ## Analytics Integrations Additionally, there is a variety of [Analytics Integrations](https://developers.cloudflare.com/fundamentals/data-products/analytics-integrations) to use. Cloudflare is working hard to integrate more things, and make it easier to connect to their Logpush. > *UPDATE DECEMBER 2024: There’s now a [Grafana data source plugin](https://grafana.com/grafana/plugins/grafana-cloudflare-datasource/) ([docs](https://grafana.com/docs/plugins/grafana-cloudflare-datasource/latest/)) available.* ### New Relic Additionally, check out [Get full observability into your Cloudflare logs with New Relic](https://blog.cloudflare.com/announcing-the-new-relic-direct-log-integration/) to easily Logpush your logs to a New Relic Dashboard. ![New Relic Dashboard Example](https://davidtofan.com/cdn-cgi/image/onerror=redirect,width=1440,height=817,format=webp/_astro/logpush-newrelic-dashboard.BY4hdnY-.png) ### Sentinel One Check out how to Logpush to SentinelOne Scalyr Dataset by using [cehrig/cloudflare-logpush-sentinelone-scalyr-dataset](https://github.com/cehrig/cloudflare-logpush-sentinelone-scalyr-dataset). --- ## Disclaimer This guide was inspired by a colleague – all crediting goes to him. 🤓 Educational purposes only, and this blog post does not necessarily reflect the opinions of Cloudflare. There are many more aspects to Cloudflare and its products and services – this is merely a brief educational intro. Properly inform yourself, keep learning, keep testing, and feel free to share your learnings and experiences as I do. Hope it was helpful! Images are online and publicly accessible. ```json {"@context":"https://schema.org","@type":"BlogPosting","headline":"Cloudflare Logpush","description":"Overview of Log and Analytics products at Cloudflare.","image":"https://davidtofan.com/articles/cloudflare-logpush/featured.png","datePublished":"2022-02-03T00:00:00.000Z","dateModified":"2022-02-03T00:00:00.000Z","author":{"@type":"Person","@id":"https://davidtofan.com/#person","name":"David Tofan","url":"https://davidtofan.com/","jobTitle":"Senior Customer Engineer","sameAs":["https://github.com/DavidJKTofan","https://www.linkedin.com/in/davidtofan/","https://x.com/davidjktofan"]},"mainEntityOfPage":{"@type":"WebPage","@id":"https://davidtofan.com/articles/cloudflare-logpush/"},"inLanguage":"en","keywords":"observability, cloudflare"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://davidtofan.com/"},{"@type":"ListItem","position":2,"name":"Articles","item":"https://davidtofan.com/articles/"},{"@type":"ListItem","position":3,"name":"Cloudflare Logpush","item":"https://davidtofan.com/articles/cloudflare-logpush/"}]} ```