主頁 > 知識庫 > 一次關(guān)于Redis內(nèi)存詭異增長的排查過程實戰(zhàn)記錄

一次關(guān)于Redis內(nèi)存詭異增長的排查過程實戰(zhàn)記錄

熱門標(biāo)簽:西藏教育智能外呼系統(tǒng)價格 最簡單的百度地圖標(biāo)注 地圖標(biāo)注費用 地圖標(biāo)注如何即時生效 太原營銷外呼系統(tǒng) 小紅書怎么地圖標(biāo)注店 百度商家地圖標(biāo)注怎么做 竹間科技AI電銷機(jī)器人 玄武湖地圖標(biāo)注

一、現(xiàn)象

實例名:r-bp1cxxxxxxxxxd04(主從)

問題:一分鐘內(nèi)存上漲了2G,如下圖所示:

鍵值規(guī)模:6000萬左右

內(nèi)存一分鐘增長2G.png

二、Redis內(nèi)存分析

1. 內(nèi)存組成

上圖中的內(nèi)存統(tǒng)計的是Redis的info memory命令中的used_memory屬性,例如:

redis>infomemory#Memoryused_memory:9195978072used_memory_human:8.56Gused_memory_rss:9358786560used_memory_peak:10190212744used_memory_peak_human:9.49Gused_memory_lua:38912mem_fragmentation_ratio:1.02mem_allocator:jemalloc-3.6.0 

每個屬性的詳細(xì)說明

屬性名 屬性說明
used_memory Redis 分配器分配的內(nèi)存量,也就是實際存儲數(shù)據(jù)的內(nèi)存總量
used_memory_human 以可讀格式返回 Redis 使用的內(nèi)存總量
used_memory_rss 從操作系統(tǒng)的角度,Redis進(jìn)程占用的總物理內(nèi)存
used_memory_peak 內(nèi)存分配器分配的最大內(nèi)存,代表used_memory的歷史峰值
used_memory_peak_human 以可讀的格式顯示內(nèi)存消耗峰值
used_memory_lua Lua引擎所消耗的內(nèi)存
mem_fragmentation_ratio used_memory_rss /used_memory比值,表示內(nèi)存碎片率
mem_allocator Redis 所使用的內(nèi)存分配器。默認(rèn): jemalloc

計算公式如下:

used_memory = 自身內(nèi)存+對象內(nèi)存+緩沖內(nèi)存+lua內(nèi)存used_rss = used_memory + 內(nèi)存碎片

如下圖所示:


2. 內(nèi)存分析

(1) 自身內(nèi)存:一個空的Redis占用很小,可以忽略不計

(2) kv內(nèi)存:key對象 + value對象

(3) 緩沖區(qū):客戶端緩沖區(qū)(普通 + slave偽裝 + pubsub)以及aof緩沖區(qū)(比較固定,一般沒問題)

(4) Lua:Lua引擎所消耗的內(nèi)存

3. 內(nèi)存突增常見問題

(1) kv內(nèi)存:bigkey、大量寫入

(2) 客戶端緩沖區(qū):一般常見的有普通客戶端緩沖區(qū)(例如monitor命令)或者pubsub客戶端緩沖區(qū)

三、問題排查

(1) bigkey ? 經(jīng)掃描未發(fā)現(xiàn)bigkey

Sampled 67234427 keys in the keyspace!
Total key length in bytes is 1574032382 (avg len 23.41)

Biggest string found 'CCARD_DEVICE_CARD_REF_MAP_KEY_016817000004209' has 20862 bytes
Biggest list found 'CCARD_VALID_DEVICE_TRAIN_QUEUE_KEY' has 51 items
Biggest hash found 'CCARD_VALID_DEVICE_TRAIN_MAP_KEY' has 51 fields

67234359 strings with 71767890 bytes (100.00% of keys, avg size 1.07)
67 lists with 151 items (00.00% of keys, avg size 2.25)
0 sets with 0 members (00.00% of keys, avg size 0.00)
1 hashs with 51 fields (00.00% of keys, avg size 51.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)

(2) 鍵值個數(shù)增加?未發(fā)現(xiàn)鍵值有明顯變化


(3) 客戶端緩沖區(qū)

由于內(nèi)存增上去后,長時間沒下落,如果是因為緩沖區(qū)問題,會從info clients找到明顯問題,執(zhí)行后發(fā)現(xiàn):

redis> info clients
# Clients
connected_clients:43
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
admin_clients:6
rejected_vpc_conn_count:0
close_idle_unknown_conn_count:0

執(zhí)行client中也沒有明顯的omem大于0的情況

id=80207addr=10.xx.0.4:63920fd=46name=age=624idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80215addr=10.xx.0.23:43489fd=36name=age=591idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80366addr=10.xx.0.8:59785fd=18name=age=84idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=delread=0write=0type=user
id=80356addr=10.xx.0.33:32117fd=13name=age=114idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80064addr=10.xx.59.4:53446fd=38name=age=1070idle=1070flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80276addr=10.xx.0.23:48511fd=8name=age=387idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80188addr=10.xx.0.33:16265fd=42name=age=681idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80326addr=10.xx.0.32:59779fd=16name=age=209idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80065addr=10.xx.59.4:53447fd=45name=age=1070idle=1070flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=79936addr=10.xx.0.22:10607fd=30name=age=1480idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80174addr=10.xx.0.5:60914fd=6name=age=722idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80300addr=10.xx.0.22:22757fd=48name=age=298idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80037addr=10.xx.0.5:55189fd=15name=age=1143idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80330addr=10.xx.0.8:48533fd=17name=age=199idle=10flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79896addr=10.xx.0.30:26814fd=11name=age=1616idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80299addr=10.xx.0.24:11227fd=44name=age=303idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80086addr=10.xx.0.32:52526fd=40name=age=1002idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80202addr=10.xx.0.33:16658fd=26name=age=636idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80256addr=10.xx.0.24:60496fd=19name=age=448idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79908addr=10.xx.0.29:18975fd=12name=age=1583idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80365addr=10.xx.0.29:46429fd=14name=age=85idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79869addr=10.xx.27.4:48455fd=35name=age=1700idle=1700flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80334addr=10.xx.0.23:50012fd=39name=age=189idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80041addr=10.xx.0.32:51107fd=33name=age=1132idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=79992addr=10.xx.0.22:12068fd=28name=age=1289idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80251addr=10.xx.0.30:44213fd=23name=age=468idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80006addr=10.xx.0.2:45895fd=31name=age=1242idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80321addr=10.xx.0.30:48048fd=5name=age=224idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80381addr=10.xx.0.8:13360fd=22name=age=24idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=delread=0write=0type=user
id=80200addr=10.xx.0.24:59183fd=24name=age=640idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80113addr=10.xx.0.2:52492fd=21name=age=915idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=174addr=11.216.117.242:53027fd=9name=age=281390idle=0flags=S db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=replconf read=0write=0type=admin
id=79991addr=10.xx.0.4:48412fd=25name=age=1296idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80301addr=127.0.0.1:47869fd=49name=age=291idle=261flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=strlen read=0write=0type=admin
id=80047addr=10.xx.59.4:53184fd=41name=age=1114idle=1114flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=80236addr=10.xx.0.5:62546fd=47name=age=516idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80364addr=10.xx.0.4:18794fd=7name=age=85idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80175addr=10.xx.0.4:62245fd=29name=age=718idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80336addr=10.xx.0.29:45701fd=50name=age=180idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80050addr=10.xx.59.4:53188fd=43name=age=1114idle=1114flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin
id=79765addr=10.xx.0.2:33832fd=37name=age=2027idle=177flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=info read=0write=0type=user
id=80170addr=10.xx.0.2:57853fd=20name=age=728idle=24flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user
id=80390addr=127.0.0.1:49449fd=27name=age=0idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=client read=0write=0type=admin

四、揪出元兇

常用的幾招都用了,還是不行,同事@徑遠(yuǎn)幫忙一起分析,懷疑是不是因為Redis的kv哈希表做了 rehash。

1. Redis的kv存儲結(jié)構(gòu)

如下圖所示,Redis的所有kv保存在dict中,其中ht對應(yīng)兩個哈希表ht[0]和ht[1],平時一個空閑,一個用于存儲數(shù)據(jù),只有當(dāng)需要rehash時,ht[1]才會用到。


2. Redis的字典rehash

為了保證哈希表的負(fù)載,當(dāng)哈希表的元素個數(shù)等于哈希表槽數(shù)時候,會進(jìn)行rehash擴(kuò)容。擴(kuò)容后h[1]的容量等于第一個大于等于ht[0].size*2的2n,例如hash表的初始化容量是4,那么下一次擴(kuò)容就是8,以此類推。

3. 測試

(1) 測試方法

先批量寫入到rehash閾值附近,然后在逐條去寫,觀察內(nèi)存變化

// 為每個鍵設(shè)置1天過期時間
int expireTime = 60 * 60 * 24;
// rehash閾值 - 50為了方便觀察rehash內(nèi)存變化
int rehashThreshold = (int) Math.pow(2, 25) - 50;

// 1.批量寫入:pipeline批量寫入,由于是本機(jī)測試,這里用10000,實際生產(chǎn)不要這么用
Pipeline pipeline = jedis.pipelined();
pipeline = jedis.pipelined();
for (int i = 0; i  rehashThreshold; i++) {
  pipeline.setex(String.valueOf(i), expireTime, String.valueOf(i));
  if (i % 10000 == 0) {
    pipeline.sync();
  }
}
pipeline.sync();

// 2.等待寫增量
TimeUnit.SECONDS.sleep(5);
for (int i = rehashThreshold; i  rehashThreshold + 200; i++) {
  jedis.setex(String.valueOf(i), expireTime, String.valueOf(i));
  TimeUnit.SECONDS.sleep(1);
}

(2) 開始測試

(a) 當(dāng)閾值=215=32768,從下面可以看出到key的個數(shù)為32769時,內(nèi)存漲了一些,但是還不明顯。

​keys       mem      clients blocked requests            connections32766      4.69M    3       0       32797 (+2)          4
32767      4.69M    3       0       32799 (+2)          4
32768      4.69M    3       0       32801 (+2)          4
32769      5.44M    3       0       32803 (+2)          4

(b) 當(dāng)閾值=220=1048576,從下面可以看出到key的個數(shù)為1048577時,內(nèi)存漲了32M。因為rehash會擴(kuò)容,所以新的哈希表中的槽位變?yōu)榱?21 * 2(因為每個key都設(shè)置了過期時間,expires表),指針為8個字節(jié),221 ? 2 ? 8 = 225 = 32MB。

​keys       mem      clients blocked requests            connections1048574    128.69M  3       0       3364129 (+2)        16
1048575    128.69M  3       0       3364131 (+2)        16
1048576    128.69M  3       0       3364133 (+2)        16
1048577    160.69M  3       0       3364135 (+2)        16
1048578    160.69M  3       0       3364137 (+2)        16

(c) 當(dāng)閾值=226=67108864,從下面可以看出到key的個數(shù)為67108865時,內(nèi)存漲了2GB。因為rehash會擴(kuò)容,所以新的哈希表中的槽位變?yōu)榱?27 * 2(因為每個key都設(shè)置了過期時間,expires表),指針為8個字節(jié),227 ? 2 ? 8 = 231 = 2GB。

​keys       mem      clients blocked requests            connections67108862   9.70G    3       0       70473683 (+2)       18
67108863   9.70G    3       0       70473685 (+2)       18
67108864   9.70G    3       0       70473687 (+2)       18
67108865   11.70G   3       0       70473689 (+2)       18
67108866   11.70G   3       0       70473691 (+2)       18
67108867   11.70G   3       0       70473693 (+2)       18

回過來看r-bp1c15fd9b142d04的key和內(nèi)存變化圖,可以發(fā)現(xiàn)上面的規(guī)則是正確的:

 

4. 后續(xù)觀察

17點時,rehash結(jié)束,內(nèi)存降了增加的2G的一半。


五、總結(jié)

由于哈希表的特性,Redis 中鍵值數(shù)量大,不會對存取造成性能影響,但是會出現(xiàn)本文提到的問題。控制鍵個數(shù)有幾個建議:無用的鍵值設(shè)置過期時間或者定期刪除。優(yōu)化鍵值設(shè)計:例如可以使用 ziplist hash合并優(yōu)化部分字符串類型。未來改進(jìn):內(nèi)核層面支持 rehash 的審計日志以及增強(qiáng) rehash 的速度。

好了,以上就是這篇文章的全部內(nèi)容了,希望本文的內(nèi)容對大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價值,如果有疑問大家可以留言交流,謝謝大家對腳本之家的支持。

您可能感興趣的文章:
  • redis內(nèi)存空間效率問題的深入探究
  • redis 限制內(nèi)存使用大小的實現(xiàn)
  • redis 使用lettuce 啟動內(nèi)存泄漏錯誤的解決方案
  • 淺談內(nèi)存耗盡后Redis會發(fā)生什么
  • 淺談redis內(nèi)存數(shù)據(jù)的持久化方式
  • 內(nèi)存型數(shù)據(jù)庫Redis持久化小結(jié)
  • 降低PHP Redis內(nèi)存占用
  • Redis教程(十四):內(nèi)存優(yōu)化介紹
  • 詳解Redis瘦身指南

標(biāo)簽:景德鎮(zhèn) 香港 廣東 唐山 贛州 澳門 林芝 揚州

巨人網(wǎng)絡(luò)通訊聲明:本文標(biāo)題《一次關(guān)于Redis內(nèi)存詭異增長的排查過程實戰(zhàn)記錄》,本文關(guān)鍵詞  一次,關(guān)于,Redis,內(nèi)存,詭異,;如發(fā)現(xiàn)本文內(nèi)容存在版權(quán)問題,煩請?zhí)峁┫嚓P(guān)信息告之我們,我們將及時溝通與處理。本站內(nèi)容系統(tǒng)采集于網(wǎng)絡(luò),涉及言論、版權(quán)與本站無關(guān)。
  • 相關(guān)文章
  • 下面列出與本文章《一次關(guān)于Redis內(nèi)存詭異增長的排查過程實戰(zhàn)記錄》相關(guān)的同類信息!
  • 本頁收集關(guān)于一次關(guān)于Redis內(nèi)存詭異增長的排查過程實戰(zhàn)記錄的相關(guān)信息資訊供網(wǎng)民參考!
  • 推薦文章