ENXF NET
Administrator
Staff member
Administrator
Moderator
+Lifetime VIP+
S.V.I.P.S Member
S.V.I.P Member
V.I.P Member
Collaborate
Registered
- Joined
- Nov 13, 2018
- Messages
- 25,964
- Points
- 823
We have got a forum site that has a lot of threads/posts. Thus, if we go to the forum listing page, it has a "lot of" pages (In paging) of threads.
Now some crawlers access these deep forum pages, like 60000th page, 70000th page etc.
"GET /forum/latest-news/page-64099 HTTP/1.1"
"GET /forum/latest-news/page-65000 HTTP/1.1"
This creates a heavy load on the db because MySQL offset queries will be heavy to fetch this kind of content. And eventually, the site goes down in front-end for a...
Read more
Continue reading...
Now some crawlers access these deep forum pages, like 60000th page, 70000th page etc.
"GET /forum/latest-news/page-64099 HTTP/1.1"
"GET /forum/latest-news/page-65000 HTTP/1.1"
This creates a heavy load on the db because MySQL offset queries will be heavy to fetch this kind of content. And eventually, the site goes down in front-end for a...
Read more
Continue reading...