Go 综合实战题详解(Web框架/RPC/K8s Controller)

2022-02-11T10:00:00+08:00 | 9分钟阅读 | 更新于 2022-02-11T10:00:00+08:00

@
综合实战题Web 框架RPC 框架分布式系统路由匹配中间件链JSON响应TCP传输JSON-RPC消息编码任务调度短链接Router Tree反射调用K8s InformerContext传递序列化延迟队列Handler封装Redis ZSetBase62编码Snowflake发号

学习目标

读完本文,你应该能够:

  • 实现一个支持路由、中间件、JSON 响应的简易 Web Framework
  • 实现 gRPC 客户端和服务端骨架代码
  • 实现分布式任务调度系统(带重试和失败告警)
  • 实现短链接服务:发号器 + Base62 + 缓存
  • 实现延迟队列:Redis ZSet 轮询版与时间轮版

一、简易 Web Framework(仿 Gin)

1.1 核心结构

应用场景

在实际项目中,这个知识点经常应用于:

  1. 典型场景一:说明具体应用场景
  2. 典型场景二:说明另一个应用场景
  3. 面试要点:提醒面试中可能被问到的点
package miniengine

import (
	"encoding/json"
	"net/http"
	"strings"
)

type Context struct {
	writer   http.ResponseWriter
	req      *http.Request
	params   map[string]string
	status   int
	handler  HandlerFunc
}

type HandlerFunc func(*Context)
type Middleware func(HandlerFunc) HandlerFunc

type Router struct {
	patterns []string
	handlers []HandlerFunc
	mw       []Middleware
}

type Engine struct {
	router *Router
}

func New() *Engine {
	return &Engine{
		router: &Router{
			patterns: make([]string, 0),
			handlers: make([]HandlerFunc, 0),
		},
	}
}

1.2 路由注册

func (e *Engine) GET(pattern string, handler HandlerFunc) {
	e.router.AddRoute("GET", pattern, handler)
}

func (e *Engine) POST(pattern string, handler HandlerFunc) {
	e.router.AddRoute("POST", pattern, handler)
}

func (r *Router) AddRoute(method, pattern string, handler HandlerF

### 应用场景

在实际项目中,这个知识点经常应用于:

1. **典型场景一**:说明具体应用场景
2. **典型场景二**:说明另一个应用场景
3. **面试要点**:提醒面试中可能被问到的点

unc) {
	r.patterns = append(r.patterns, method+"."+pattern)
	r.handlers = append(r.handlers, handler)
}

1.3 中间件链

func (e *Engine) Use(mw ...Middleware) {
	e.router.mw = append(e.router.mw, mw...)
}

func (r *Router) Chain(handler HandlerFunc) HandlerFunc {
	for i := len(r.mw) - 1; i >= 0; i-- {
		handler = r.mw[i](handler)
	}
	return handler
}

Web Framework 请求流:Router 匹配 → 中间件链(洋葱模型)→ Handler → 响应

请求到达ServeHTTP
<rect x="200" y="170" width="150" height="60" rx="10"/>
<text x="275" y="194" text-anchor="middle" fill="#1a1a1a" font-size="12" font-weight="700" stroke="none">Router 匹配</text>
<text x="275" y="214" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">findHandler</text>
<rect x="370" y="170" width="150" height="60" rx="10"/>
<text x="445" y="194" text-anchor="middle" fill="#1a1a1a" font-size="12" font-weight="700" stroke="none">Mw1(日志)</text>
<text x="445" y="214" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">外层</text>
<rect x="540" y="170" width="150" height="60" rx="10"/>
<text x="615" y="194" text-anchor="middle" fill="#1a1a1a" font-size="12" font-weight="700" stroke="none">Mw2(鉴权)</text>
<text x="615" y="214" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">内层</text>
<rect x="710" y="170" width="220" height="60" rx="10"/>
<text x="820" y="194" text-anchor="middle" fill="#1a1a1a" font-size="12" font-weight="700" stroke="none">Handler(业务 + JSON)</text>
<text x="820" y="214" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">ctx.JSON 写回</text>
<rect x="710" y="310" width="220" height="60" rx="10"/>
<text x="820" y="334" text-anchor="middle" fill="#1a1a1a" font-size="12" font-weight="700" stroke="none">响应写回 writer</text>
<text x="820" y="354" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">Header + Body</text>
Chain 从外到内包裹 Handler洋葱模型返回:逆序经过 Mw2 → Mw1

看图要点:中间件链的本质是"函数套娃"——Chain 把 mw[i] 从外到内逐层包裹 Handler(代码里是逆序 for 拼接),请求进入顺序 Mw1→Mw2→Handler,返回时逆序穿过(洋葱模型),所以前置/后置逻辑写在同一个中间件里。面试官追问:① 多个中间件 panic 怎么恢复(recover 在中层);② 鉴权中间件失败如何短路(不调 next);③ 和 Gin 的 c.Next() 区别;④ 中间件顺序为什么敏感(如 recover 必须最外层)。

1.4 HTTP 服务器

func (e *Engine) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	ctx := &Context{writer: w, req: r, status: 200}
	
	// 查找匹配的路由
	handler := e.findHandler(ctx)
	if h

### 应用场景

在实际项目中,这个知识点经常应用于:

1. **典型场景一**:说明具体应用场景
2. **典型场景二**:说明另一个应用场景
3. **面试要点**:提醒面试中可能被问到的点

andler == nil {
		http.NotFound(w, r)
		return
	}
	
	// 应用中间件链
	chainHandler := e.router.Chain(handler)
	chainHandler(ctx)
}

func (e *Engine) Run(addr string) error {
	return http.ListenAndServe(addr, e)
}

1.5 JSON 响应

func (ctx *Context) JSON(status int, data interface{}) {
	ctx.writer.Header().Set("Content-Type", "application/json")
	ctx.writer.WriteHeader(status)
	json.NewEncoder(ctx.writer).Encode(data)
}

func (ctx *Context) String(

### 应用场景

在实际项目中,这个知识点经常应用于:

1. **典型场景一**:说明具体应用场景
2. **典型场景二**:说明另一个应用场景
3. **面试要点**:提醒面试中可能被问到的点

status int, format string, values ...interface{}) {
	ctx.writer.Header().Set("Content-Type", "text/plain")
	ctx.writer.WriteHeader(status)
	ctx.writer.Write([]byte(fmt.Sprintf(format, values...)))
}

func (ctx *Context) Body(status int, body []byte) {
	ctx.writer.WriteHeader(status)
	ctx.writer.Write(body)
}

二、gRPC 骨架实现

2.1 服务端

type Server struct {
	addr     string
	service  map[string]interface{}
	codec    Codec
	listener net.Listener
}

func NewServer(addr string, codec Codec) *Server {
	return &Server{
		addr:    addr,
		service: make(map[string]interface{}),
		codec:   codec,
	}
}

func (s *Server) Register(name string, svc interface{}) {
	s.service[name] = svc
}

func (s *Server) Serve() error {
	listener, err := net.Listen("tcp", s.addr)
	if err != nil {
		return err
	}
	s.listener = listener
	
	for {
		conn, err := listener.Accept()
		if err != nil {
			continue
		}
		go s.handleConn(conn)
	}
}

func (s *Server) handleConn(conn net.Conn) {
	defer conn.Close()
	
	// 读取请求头
	header := make([]byte, 4)
	if _, err := io.ReadFull(conn, header); err != nil {
		return
	}
	msgLen := binary.BigEndian.Uint32(header)
	
	// 读取消息体
	body := make([]byte, msgLen)
	if _, err := io.ReadFull(conn, body); err != nil {
		return
	}
	
	// 解码
	var req Request
	if err := s.codec.Decode(body, &req); err != nil {
		return
	}
	
	// 路由到对应服务
	svc, ok := s.service[req.Service]
	if !ok {
		return
	}
	
	// 反射调用
	resp := s.invoke(svc, req)
	
	// 编码并发送
	respBody, _ := s.codec.Encode(resp)
	respHeader := make([]byte, 4)
	binary.BigEndian.PutUint32(respHeader, uint32(len(respBody)))
	conn.Write(append(respHeader, respBody...))
}

2.2 客户端

type Client struct {
	conn    net.Conn
	codec   Codec
	timeout time.Duration
}

func Dial(addr string, codec Codec) (*Client, error) {
	conn, err := net.Dial("tcp", addr)
	if err != nil {
		return nil, err
	}
	return &Client{conn: conn, codec: codec, timeout: 5 * time.Second}, nil
}

func (c *Client) Call(service, method string, args, reply interface{}) error {
	req := Request{
		Service: service,
		Method:  method,
		Args:    args,
	}
	
	body, err := c.codec.Encode(req)
	if err != nil {
		return err
	}
	
	// 发送请求
	header := make([]byte, 4)
	binary.BigEndian.PutUint32(header, uint32(len(body)))
	if _, err := c.conn.Write(append(header, body...)); err != nil {
		return err
	}
	
	// 设置超时
	c.conn.SetDeadline(time.Now().Add(c.timeout))
	defer c.conn.SetDeadline(time.Time{})
	
	// 读取响应
	respHeader := make([]byte, 4)
	if _, err := io.ReadFull(c.conn, respHeader); err != nil {
		return err
	}
	respLen := binary.BigEndian.Uint32(respHeader)
	
	respBody := make([]byte, respLen)
	if _, err := io.ReadFull(c.conn, respBody); err != nil {
		return err
	}
	
	return c.codec.Decode(respBody, reply)
}

三、分布式任务调度系统

type Task struct {
	ID         string
	Type       string
	Payload    []byte
	Retries    int
	MaxRetries int
	ScheduledAt time.Time
	Status     string // pending/running/done/failed
}

type Scheduler struct {
	tasks    map[string]*Task
	queue    *PriorityQueue
	retryCh  chan *Task
	failCh   chan *Task
}

func (s *Scheduler) Submit(task *Task) {
	task.Status = "pending"
	s.tasks[task.ID] = task
	s.queue.Push(task)
}

func (s *Scheduler) Execute() {
	for {
		task := s.queue.Pop()
		if task == nil {
			time.Sleep(100 * time.Millisecond)
			continue
		}
		
		task.Status = "running"
		go s.runTask(task)
	}
}

func (s *Scheduler) runTask(task *Task) {
	defer func() {
		if r := recover(); r != nil {
			task.Status = "failed"
			s.handleError(task, fmt.Errorf("%v", r))
		}
	}()
	
	if err := task.Handler(); err != nil {
		task.Retries++
		if task.Retries >= task.MaxRetries {
			task.Status = "failed"
			s.failCh <- task
		} else {
			task.Status = "pending"
			s.retryCh <- task
		}
	} else {
		task.Status = "done"
	}
}

func (s *Scheduler) handleError(task *Task, err error) {
	// 发送告警
	log.Printf("Task %s failed: %v, retries: %d/%d", 
		task.ID, err, task.Retries, task.MaxRetries)
}

分布式任务调度执行流:提交 → 执行 → 重试回边 → 死信

Submit(task)写入 tasks map + 入 PriorityQueue
<rect x="60" y="130" width="260" height="56" rx="10"/>
<text x="190" y="162" text-anchor="middle" fill="#1a1a1a" font-size="12.5" font-weight="700" stroke="none">PriorityQueue(优先级堆)</text>
<rect x="60" y="230" width="260" height="56" rx="10"/>
<text x="190" y="262" text-anchor="middle" fill="#1a1a1a" font-size="12.5" font-weight="700" stroke="none">Execute:循环 Pop</text>
<rect x="60" y="330" width="260" height="56" rx="10"/>
<text x="190" y="362" text-anchor="middle" fill="#1a1a1a" font-size="12.5" font-weight="700" stroke="none">runTask(go 并发执行)</text>
<rect x="60" y="440" width="260" height="56" rx="10"/>
<text x="190" y="464" text-anchor="middle" fill="#1a1a1a" font-size="12.5" font-weight="700" stroke="none">成功 → Status=done</text>
<text x="190" y="484" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">结束</text>
<rect x="380" y="330" width="260" height="56" rx="10"/>
<text x="510" y="354" text-anchor="middle" fill="#1a1a1a" font-size="12.5" font-weight="700" stroke="none">失败:Retries++</text>
<text x="510" y="374" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">与 MaxRetries 比较</text>
<rect x="700" y="250" width="260" height="56" rx="10" stroke-dasharray="7 4"/>
<text x="830" y="274" text-anchor="middle" fill="#1a1a1a" font-size="12.5" font-weight="700" stroke="none">retryCh:重新入队</text>
<text x="830" y="294" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">稍后再次执行</text>
<rect x="700" y="120" width="260" height="56" rx="10" stroke-dasharray="7 4"/>
<text x="830" y="144" text-anchor="middle" fill="#1a1a1a" font-size="12.5" font-weight="700" stroke="none">failCh → 告警 / 死信队列</text>
<text x="830" y="164" text-anchor="middle" fill="#666" font-size="10.5" stroke="none">人工处理</text>
失败< MaxRetries≥ MaxRetries回边:retryCh 重新入 PriorityQueue

看图要点:调度系统的核心闭环是"失败自动重试"——runTask 用 goroutine 并发执行,失败且未超 MaxRetries 就进 retryCh 重新入队(回边),超过则进 failCh 转死信队列人工处理。面试官追问:① 任务怎么不丢失(持久化 + 失败重试 + 死信);② 怎么防重复执行(幂等);③ 超时 running 任务怎么兜底(定时扫描重新入队);④ 为什么用 channel 而非共享变量做重试/失败通知(解耦 + 并发安全)。


四、短链接服务

type ShortLinkService struct {
	seq     *Snowflake
	cache   *TTLCache
	base62  *Base62Encoder
}

func (s *ShortLinkService) Generate(longURL string) (string, error) {
	// 1. 生成唯一 ID
	id := s.seq.NextID()
	
	// 2. 转换为 Base62
	shortCode := s.base62.Encode(id)
	
	// 3. 存储映射
	s.cache.Set(shortCode, longURL, 30*24*time.Hour)
	
	return shortCode, nil
}

func (s *ShortLinkService) Resolve(shortCode string) (string, error) {
	url, ok := s.cache.Get(shortCode)
	if !ok {
		return "", fmt.Errorf("short code not found")
	}
	return url.(string), nil
}

type Base62Encoder struct {
	charset string
}

func NewBase62Encoder() *Base62Encoder {
	return &Base62Encoder{charset: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"}
}

func (b *Base62Encoder) Encode(n int64) string {
	if n == 0 {
		return string(b.charset[0])
	}
	var result []byte
	for n > 0 {
		result = append([]byte{b.charset[n%62]}, result...)
		n /= 62
	}
	return string(result)
}

func (b *Base62Encoder) Decode(s string) int64 {
	var result int64
	for _, c := range s {
		result = result*62 + int64(bytes.IndexByte([]byte(b.charset), byte(c)))
	}
	return result
}

五、延迟队列实现

5.1 Redis ZSet 版本

type DelayQueue struct {
	client *redis.Client
	key    string
}

func NewDelayQueue(client *redis.Client, key string) *DelayQueue {
	return &DelayQueue{client: client, key: key}
}

func (dq *DelayQueue) Enqueue(item interface{}, delay time.Duration) error {
	score := float64(time.Now().Add(delay).UnixNano())
	data, _ := json.Marshal(item)
	return dq.client.ZAdd(dq.key, redis.Z{Score: score, Member: data}).Err()
}

func (dq *DelayQueue) Dequeue(timeout time.Duration) (interface{}, error) {
	ctx, cancel := context.WithTimeout(context.Background(), timeout)
	defer cancel()
	
	for {
		select {
		case <-ctx.Done():
			return nil, ctx.Err()
		default:
			items, err := dq.client.ZRangeByScoreWithScores(ctx, dq.key, 
				&redis.ZRangeBy{Min: "0", Max: fmt.Sprintf("%d", time.Now().UnixNano())})
			if err != nil || len(items) == 0 {
				time.Sleep(100 * time.Millisecond)
				continue
			}
			
			for _, item := range items {
				dq.client.ZRem(dq.key, item.Member)
				var result interface{}
				json.Unmarshal(item.Member.([]byte), &result)
				return result, nil
			}
		}
	}
}

5.2 时间轮版本

type TimerWheel struct {
	wheels []*TickWheel
	size   int
}

type TickWheel struct {
	buckets [][]DelayedTask
	index   int
}

type DelayedTask struct {
	task     func()
	execTime int64
}

func NewTimerWheel(size int, ticks int) *TimerWheel {
	wheels := make([]*TickWheel, 0)
	prevTicks := 1
	for i := 0; i < size; i++ {
		wheels = append(wheels, &TickWheel{
			buckets: make([][]DelayedTask, ticks),
			ticks:   ticks,
		})
		prevTicks *= ticks
	}
	return &TimerWheel{wheels: wheels, size: size}
}

func (tw *TimerWheel) Schedule(task func(), delay time.Duration) {
	execTime := time.Now().Add(delay).UnixMilli()
	
	// 分配到对应的 wheel 和 bucket
	slot := execTime % int64(tw.size)
	bucket := execTime / int64(tw.size)
	
	tw.wheels[slot].buckets[bucket%int64(tw.wheels[slot].ticks)] = 
		append(tw.wheels[slot].buckets[bucket%int64(tw.wheels[slot].ticks)], 
			DelayedTask{task: task, execTime: execTime})
}

func (tw *TimerWheel) Start() {
	ticker := time.NewTicker(1 * time.Second)
	defer ticker.Stop()
	
	for range ticker.C {
		tw.tick()
	}
}

func (tw *TimerWheel) tick() {
	for _, wheel := range tw.wheels {
		idx := int(time.Now().UnixMilli()) % wheel.ticks
		for _, task := range wheel.buckets[idx] {
			task.task()
		}
		wheel.buckets[idx] = nil
	}
}

六、K8s Controller 简化实现

type Controller struct {
	informer cache.Informer
	workqueue workqueue.RateLimitingInterface
	handler  func(interface{}) error
}

func NewController(informer cache.Informer, handler func(interface{}) error) *Controller {
	c := &Controller{
		informer: informer,
		workqueue: workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()),
		handler:  handler,
	}
	
	// 添加事件处理器
	c.informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
		AddFunc:    c.enqueue,
		UpdateFunc: func(_, newObj interface{}) { c.enqueue(newObj) },
		DeleteFunc: c.enqueue,
	})
	
	return c
}

func (c *Controller) enqueue(obj interface{}) {
	key, _ := cache.MetaNamespaceKeyFunc(obj)
	c.workqueue.Add(key)
}

func (c *Controller) Run(workerCount int, stopCh <-chan struct{}) {
	defer c.workqueue.ShutDown()
	
	go c.informer.Run(stopCh)
	if !cache.WaitForCacheSync(stopCh, c.informer.HasSynced) {
		klog.ErrorS(nil, "Failed to wait for caches to sync")
		return
	}
	
	for i := 0; i < workerCount; i++ {
		go wait.Until(c.worker, time.Second, stopCh)
	}
	
	<-stopCh
}

func (c *Controller) worker() {
	for c.processNextItem() {
	}
}

func (c *Controller) processNextItem() bool {
	key, quit := c.workqueue.Get()
	if quit {
		return false
	}
	defer c.workqueue.Done(key)
	
	if err := c.handler(key); err != nil {
		c.workqueue.AddRateLimit(key)
		klog.ErrorS(err, "Failed to reconcile", "key", key)
		return true
	}
	
	c.workqueue.Forget(key)
	return true
}

七、自测题

面试官
你的 Web Framework 如何实现路由参数解析(如 `/users/:id`)?
面试官
候选人

使用 Radix Tree(前缀树)或简单的字符串匹配:

  1. 将路由模板编译成正则表达式:/users/(?P<id>[^/]+)
  2. 请求时匹配正则,提取命名参数
  3. 将参数存入 Context.Params map

    进阶方案是使用 Radix Tree,支持通配符和优先级。
面试官
分布式任务调度如何保证任务不丢失?
面试官
候选人

关键设计:

  1. 持久化:任务提交前写入数据库/Redis
  2. 幂等性:同一任务多次执行结果一致
  3. 状态机:pending → running → done/failed
  4. 超时检测:定时扫描超时未完成的 running 任务,重新入队
  5. 死信队列:超过最大重试次数的任务进入死信队列人工处理

八、本章小结

复习提示:
  • Web Framework 核心是路由匹配 + 中间件链 + Context 传递
  • RPC 框架关键是编解码 + 反射调用 + 连接复用
  • 短链接的本质是 ID 生成(Snowflake)+ 编码转换(Base62)+ 缓存存储
  • 延迟队列Redis ZSet 最简单,时间轮性能更好
  • 下一篇将讲解易错点与陷阱,覆盖并发陷阱和内存陷阱的高频面试题

参考

  • Gin 源码:https://github.com/gin-gonic/gin
  • Kitex RPC 源码:https://github.com/cloudwego/kitex
  • Kubernetes Controller 模式:https://kubernetes.io/docs/concepts/extend-kubernetes/client-libraries/
About Me

没什么想介绍的,一个很大众的码农…

喜欢代码,车,马,真的是 🐎

讨厌别人让我给自己的代码写注释 最厌烦别人的程序没有写注释

目标

学AI,加油!加油!