opentelemetry全链路初探--埋点与jaeger

前言

某天一位业务研发老哥跑来咨询

研发老哥：我的服务出现了504，但是不太清楚是哪个环节报错，每次请求需要访问4个微服务、2个数据库、1个redis、1个消息队列。。。

苦逼运维：停停停，不要再说了，目前不支持链路追踪，只能手动帮你一个服务一个服务的排查了

先请老哥大概描述了一下业务逻辑以及访问方式，10分钟过去了。再逐级排查每个服务以及对应访问的资源层，终于在半小时之后完成了故障定位。。。

这效率也太低了，于是，关于链路建设项目提上了议程，目标只有一个，快速定位问题，提高稳定性。而链路建设，OpenTelemetry是目前行业热点，那本运维就来研究研究

环境准备

组件 版本

操作系统 Ubuntu 22.04.4 LTS

opentelemetry-sdk 1.35.0

安装

首先先简单说一下OpenTelemetry的数据采集流程，然后先跑起来再去讨论细节

OpenTelemetry就是在代码中埋入采集点进行数据采集，opentelemetry-sdk

再通过固定的协议将数据上传至某个地方进行数据展示，jaeger UI

安装OpenTelemetry-sdk

pip3 install opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-api

安装数据展示jaeger UI

docker pull docker.m.daocloud.io/jaegertracing/all-in-one:latest

docker run -d --name jaeger \

-e COLLECTOR_OTLP_ENABLED=true \

-p 16686:16686 \

-p 4317:4317 \

-p 4318:4318 \

docker.m.daocloud.io/jaegertracing/all-in-one:latest

docker启动之后访问：http://127.0.0.1:16686

watermarked-first_1

第一个例子

web服务

首先先准备一个web服务，这里我们用tornado来实现，安装tornado：pip3 install tornado

import tornado.httpserver as httpserver

import tornado.web

from tornado.ioloop import IOLoop

class TestFlow(tornado.web.RequestHandler):

def get(self):

self.finish('hello world')

def applications():

urls = []

urls.append([r'/', TestFlow])

return tornado.web.Application(urls)

def main():

app = applications()

server = httpserver.HTTPServer(app)

server.bind(10000, '0.0.0.0')

server.start(1)

IOLoop.current().start()

if __name__ == "__main__":

try:

main()

except KeyboardInterrupt as e:

IOLoop.current().stop()

finally:

IOLoop.current().close()

检查是否能够正常访问：

watermarked-first_2

添加埋点

import tornado.httpserver as httpserver

import tornado.web

from tornado.ioloop import IOLoop

from opentelemetry import trace

from opentelemetry.sdk.trace import TracerProvider

from opentelemetry.sdk.resources import SERVICE_NAME, Resource

from opentelemetry.sdk.trace.export import BatchSpanProcessor

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(

TracerProvider(resource=Resource.create({SERVICE_NAME: "s1"}))

)

tracer = trace.get_tracer(__name__)

span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))

trace.get_tracer_provider().add_span_processor(span_processor)

class TestFlow(tornado.web.RequestHandler):

def get(self):

views()

self.finish('hello world')

def views():

span = tracer.start_span("s1-span")

span.end()

def applications():

urls = []

urls.append([r'/', TestFlow])

return tornado.web.Application(urls)

def main():

app = applications()

server = httpserver.HTTPServer(app)

server.bind(10000, '0.0.0.0')

server.start(1)

IOLoop.current().start()

if __name__ == "__main__":

try:

main()

except KeyboardInterrupt as e:

IOLoop.current().stop()

finally:

IOLoop.current().close()

再次访问 curl http://localhost:10000 ，打开jaeger UI查看

watermarked-first_3

watermarked-first_4

已经有数据了，刚才的埋点已经上报至jaeger UI了

埋点数据属性

丰富一下埋点数据的属性

def views():

span = tracer.start_span("s1-span")

span.set_attribute("name", "wilson")

span.set_attribute("addr", "cd")

span.end()

watermarked-first_5

增加数据库访问追踪

def views():

span = tracer.start_span("s1-span")

span.set_attribute("name", "wilson")

span.set_attribute("addr", "cd")

ctx = trace.set_span_in_context(span)

get_db(ctx)

span.end()

def get_db(parent_ctx):

span = tracer.start_span("s1-span-db", context=parent_ctx)

span.end()

watermarked-first_6

增加跨服务追踪

增加第二个web服务：s2.py

import tornado.httpserver as httpserver

import tornado.web

from tornado.ioloop import IOLoop

from opentelemetry import trace

from opentelemetry.sdk.trace import TracerProvider

from opentelemetry.sdk.resources import SERVICE_NAME, Resource

from opentelemetry.sdk.trace.export import BatchSpanProcessor

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

trace.set_tracer_provider(

TracerProvider(resource=Resource.create({SERVICE_NAME: "s2"}))

)

tracer = trace.get_tracer(__name__)

span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))

trace.get_tracer_provider().add_span_processor(span_processor)

class TestFlow(tornado.web.RequestHandler):

def get(self):

ctx = TraceContextTextMapPropagator().extract(self.request.headers)

span = tracer.start_span("s2-span", context=ctx)

span.end()

self.finish('hello world')

def applications():

urls = []

urls.append([r'/', TestFlow])

return tornado.web.Application(urls)

def main():

app = applications()

server = httpserver.HTTPServer(app)

server.bind(20000, '0.0.0.0')

server.start(1)

IOLoop.current().start()

if __name__ == "__main__":

try:

main()

except KeyboardInterrupt as e:

IOLoop.current().stop()

finally:

IOLoop.current().close()

修改s1.py

from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

import requests

def views():

span = tracer.start_span("s1-span")

span.set_attribute("name", "wilson")

span.set_attribute("addr", "cd")

ctx = trace.set_span_in_context(span)

get_db(ctx)

headers = {}

TraceContextTextMapPropagator().inject(headers, context=ctx)

requests.get("http://localhost:20000", headers=headers)

span.end()

watermarked-first_7

改造进k8s

jaeger

编排文件：

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: jaeger

name: jaeger

namespace: default

spec:

replicas: 1

selector:

matchLabels:

app: jaeger

template:

metadata:

labels:

app: jaeger

spec:

containers:

- image: docker.m.daocloud.io/jaegertracing/all-in-one:latest

imagePullPolicy: Always

name: jaeger

dnsPolicy: ClusterFirst

restartPolicy: Always

---

apiVersion: v1

kind: Service

metadata:

labels:

app: jaeger-service

name: jaeger-service

namespace: default

spec:

ports:

- name: port-4317

port: 4317

protocol: TCP

targetPort: 4317

- name: port-4318

port: 4318

protocol: TCP

targetPort: 4318

- name: port-16686

port: 16686

protocol: TCP

targetPort: 16686

selector:

app: jaeger

type: NodePort

s2

1）制作镜像

由于在k8s集群中通过svc访问jaeger，需要改造一下s2.py

s2.py

...

import os

JAEGER_ADDR=os.environ.get('JAEGER_ADDR')

...

span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=JAEGER_ADDR))

...

Dockerfile

FROM python:3.8

WORKDIR /opt

RUN pip3 install tornado opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp -i https://pypi.tuna.tsinghua.edu.cn/simple

ADD s2.py /opt

CMD python3 s2.py

2）编排文件

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: s2

name: s2

namespace: default

spec:

replicas: 1

selector:

matchLabels:

app: s2

template:

metadata:

labels:

app: s2

spec:

containers:

- env:

- name: JAEGER_ADDR

value: http://jaeger-service:4318/v1/traces

image: s2:v1

imagePullPolicy: Always

name: s2

dnsPolicy: ClusterFirst

restartPolicy: Always

---

apiVersion: v1

kind: Service

metadata:

labels:

app: s2-service

name: s2-service

namespace: default

spec:

ports:

- name: s2-port

port: 20000

protocol: TCP

targetPort: 20000

selector:

app: s2

type: NodePort

s1

1）制作镜像

由于在k8s集群中通过svc访问s2与jaeger，需要改造一下s1.py

s1.py

...

import os

S2_ADDR=os.environ.get('S2_ADDR')

JAEGER_ADDR=os.environ.get('JAEGER_ADDR')

...

span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=JAEGER_ADDR))

...

def views():

span = tracer.start_span("s1-span")

span.set_attribute("name", "wilson")

span.set_attribute("addr", "cd")

ctx = trace.set_span_in_context(span)

get_db(ctx)

headers = {}

TraceContextTextMapPropagator().inject(headers, context=ctx)

requests.get(S2_ADDR, headers=headers)

span.end()

...

Dockerfile:

FROM python:3.8

WORKDIR /opt

RUN pip3 install tornado opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp -i https://pypi.tuna.tsinghua.edu.cn/simple

ADD s1.py /opt

CMD python3 s1.py

2）编排文件

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: s1

name: s1

namespace: default

spec:

replicas: 1

selector:

matchLabels:

app: s1

template:

metadata:

labels:

app: s1

spec:

containers:

- env:

- name: S2_ADDR

value: http://s2-service:20000

- name: JAEGER_ADDR

value: http://jaeger-service:4318/v1/traces

image: s1:v1

imagePullPolicy: Always

name: s1

dnsPolicy: ClusterFirst

restartPolicy: Always

---

apiVersion: v1

kind: Service

metadata:

labels:

app: s1-service

name: s1-service

namespace: default

spec:

ports:

- name: s1-port

port: 10000

protocol: TCP

targetPort: 10000

selector:

app: s1

type: NodePort

查看结果

▶ kubectl get pod -owide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

jaeger-6669cd7c4-4pl5j 1/1 Running 0 7m31s 10.244.0.236 minikube <none> <none>

s1-5c569c5b4b-lctzq 1/1 Running 0 73s 10.244.0.237 minikube <none> <none>

s2-5bb648dcdf-mlnbj 1/1 Running 0 61s 10.244.0.238 minikube <none> <none>

▶ kubectl get svc

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

jaeger-service NodePort 10.106.13.217 <none> 4317:31891/TCP,4318:31997/TCP,16686:31002/TCP 5m49s

s1-service NodePort 10.102.25.195 <none> 10000:32376/TCP 4m23s

s2-service NodePort 10.103.114.198 <none> 20000:30032/TCP 3m40s

进行数据测试：

访问s1服务

▶ curl http://192.168.49.2:32376

hello world%

查看jaeger日志，访问：http://192.168.49.2:31002/

watermarked-first_10

总结

在第一个例子中，我们主要采集了业务服务的trace记录，即一个完整的请求需要经过的路径，包括读取数据库、跨服务请求等等

在整个跟踪过程中trace_id与span_id发挥了决定性的作用，前者为请求链路的唯一标识，串联了整个访问步骤；而后者则是链路上每一次不同的具体操作的标识

watermarked-first_8

采集：通过嵌入代码埋点，采集重点监控的流程，比如数据库读写速度、下游服务速度等

处理：opentelemetry-sdk对数据进行处理：过滤、缓存、合并

导出：将处理过的数据，通过固定的协议（otlp协议、grpc协议、http协议等）发送到后端系统，比如jaeger