Bug #15128
closedShortcut Handling can cause SEO problems and should be improved
100%
Description
This issue is classified as "major" not for technical malfunctioning but as misconception:
- Shortcuts simply output the content of another page under its PID. Therefore it creates two identical pages available under different URLs. This is generally bad practice because search engines "punish" so called "duplicate content" with lower rankings.
Solution: Incoming visitors should be redirected to the page to which the shortcut is pointing to (this would technically apply to "hardlinked" shortcuts as well as "show first subpage"-shortcuts.
- In this context the type of redirection is crucial. As TYPO3 uses 302 redirection (temporarily moved) a site can be exposed to the hijacking phenomenon (i.e. in some search engines the content of a page is shown for the linked page instead for the linking page).
The proper redirection mode to apply is 301 (permanently moved) as it leads search engines to spider a page in a correct and reliable manner.
If you need more input just ask ( jh@venetowebdesign.com )
(issue imported from #M1692)
Files
Updated by Sebastian Kurfuerst almost 19 years ago
Hi,
a patch for this issue would be welcome. Greets, Sebastian
Updated by Stefano Cecere over 18 years ago
i just got into the bugtracker to propose the same problem i'm having with shortcut pages (i don't go technical, but it's quite important to solve it).
i have a proposal:
add a checkbox in the page proporties (or eventually a TS option) to give make a shortcut page to go straight to the linked page.
(the same concept could work with external URL pages)
i think i could provide a patch to make it work always, leaving to you to add the checkbox (for backward compatibility) ok?
or do you think it's better and easier to add a new line in the "Shortcut mode" selector?
something like "direct link".. or "stealth link" :)
since i think that "random link" or other options are evaluated when loading the shortcut id page, and then reloading the new one (right?) the checkbox mode couldn't easily work with these options...
Updated by jholzmann no-lastname-given over 18 years ago
Ciao Stefano, due to the relative complexity of this issue (I could imagine that an additional option in the page header might confuse people) I'd consider it best to introduce a new TS CONFIG options, for example
config.resolveShortcuts=1
config.resolveOutboundLink=1
to do it and to keep backwards compatibility. (as it exists already as config option for shortcuts in realUrl).
So everyone can decide if he want to link directly on onther URLs by this pagetype or if he wants to use the redirect mode (which is the way T3 is working right now).
saluti, Jan
Updated by Martin Kutschker over 18 years ago
There has always been the possibilty to acces the same page with differnet URLs (or at leats query arguments).
How about adding new page type "internal link"?
Perhaps this request should be discussed on dev list.
Updated by old_hkdennis2k over 18 years ago
I think there are at least three type of "link"
1. reference link
e.g.
URL build by typolink() or TMENU are pointing to the target page.
2. temporary redirect
may change after create (302, temporarily moved)
e.g.
like "random page" or "first page" under the page.
3. alias shortcut
(303, permanently moved)
used for redirect counter
e.g. used in newsletter
---
Is the last one in same category as others?
4. countdown redirect
like downloading in download.com / sf.net.
you'll have an intro page before redirect to the target
---
I don't think using typoscript's "config" is an good idea, it affect all links on one page. And it should be all pages using that shortcut.
Updated by jholzmann no-lastname-given over 18 years ago
I am not sure if it makes sense to add another link type (which may be confusing).
From my point of view it is all about the overall integrity of the whole linkstructure of a website:
- Basically every page in a site has to exist only once (now with shortcuts they exist twice) unless you tell the system to do so.
- The correctness of a URL must be assured by verification, i.e. it may not visualize the page if you mistype the URL (which is actually possible with simulateStaticDocuments).
=> It is essential to have the option to configure a site in TYPO3 (with simulatestaticdocuments) in the same way and behavious as it were if you'd create it with static files in Dreamweaver, wich means:
- you mistype the URL, you get a 404.
Do not mix up the header status codes and use them apprioriately:
302 / moved temporarily -for random shortcut
301/ moved permanently - for first subpage shortcut {but here comes in what i meant with resolve shortcuts: It should link directly to the according URL without redirection; a 301 redirect should be there ONLY if you hit the shortcut page for any reason, for examply by opening it from the backend}
Have a look here for the status codes:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
greetz
Jan
Updated by Martin Kutschker over 18 years ago
Sorry Jan, but I disagree. I see no point in adding HTTP redirects to the one-and-only URL. And if you start it here you have to add it everywhere in TYPO3.
eg TYPO3 with multi-lang and Real URL. These URLs are the same:
domain/index.php?id=123
domain/index.php?id=123&L=0
domain/seitenenpfad.html
domain/lang/pagepath.html
More examples can be constructed. And beware the query arguments with default values (eg L=0 and no L at all are the same).
Updated by Oliver Klee over 18 years ago
I don't think that we are forced to decided between "do it in all possible places" and "don't do it at all". I think we should take the first steps. This will solve some of the aforementioned problems (like prevent some Google page penalties), and we can improve on it later.
In other words: I'll be happy if some of Jan's proposals get implemented.
Updated by jholzmann no-lastname-given over 18 years ago
@ Martin
These URLs are the same:
domain/index.php?id=123
domain/index.php?id=123&L=0
domain/seitenenpfad.html
domain/lang/pagepath.html
Sure, and therfore you have a 4fold duplicate content problem if this page is somehow indexed by a spider. Of course the mainstream will not bother, but if you depend on SErankings you might have a problem.
Don't get me wrong, I didn't want to suggest a change which introduces hidden changes by default for everybody. It's only that I'd be happy to solve these issues without unelegantly dirtyHotfixing the sourcecode.
Therefore I suggested to solve it optionally by a CONFIG item for those who are aware of that problem.
cheers
Jan
Updated by Martin Kutschker over 18 years ago
Ok, but to get it right:
the problem is only with spidering.
My examples are valid URLs but are no problem as long as only one link of it's kind is generated in TYPO3. Or if not, a proper HTTP redirect to the "correct" URL is returned.
This affects not only shortcuts bu also all pages of default language, which get a L=0 by a language switch menu.
BTW, the L=0 problem also affects the indexed search. Pages may be indexed twice: once for no L and another time for L=0. But it mighy be that this problem "only" occurs when you have a cHash present.
PS: Are there any fatcs on SE ranking? Too me most claims are mythical. How do I prove that Google ranks me lower because of some standard TYPO3 shortcut pages?
Updated by Oliver Klee over 18 years ago
From http://www.google.de/webmasters/guidelines.html (in German):
Qualitätsrichtlinien – Konkrete Empfehlungen:[snip]
- Vermeiden Sie Cloaking oder irreführende Umleitungen.
[snip] - Erstellen Sie keine doppelten Seiten, Subdomänen oder Domänen, die im Grunde denselben Inhalt haben.
- Vermeiden Sie den Einsatz von "Brückenseiten" (Doorway-Seiten), die speziell für Suchmaschinen erstellt werden, oder andere "Cookie-Cutter-Techniken" wie z.B. Partnerprogramme mit keinem oder nur wenig eigenem Inhalt.
In diesen Qualitätsrichtlinien sind die gängigen irreführenden oder manipulativen Praktiken enthalten. Darüber hinaus kann es sein, dass Google negativ auf andere, hier nicht aufgeführte Verhaltensweisen reagiert (z.B. Täuschung von Nutzern durch Registrierung der absichtlich falsch buchstabierten Namen bekannter Websites). Falls ein bestimmtes irreführendes Verfahren hier nicht aufgelistet ist, lässt dies nicht den Schluss zu, dass Google dieses toleriert.
Updated by jholzmann no-lastname-given over 18 years ago
the problem is only with spidering.
Yes, if you put it this way it is kind of an "outbound" problem.
PS: Are there any fatcs on SE ranking? Too me most claims are mythical. How do I prove that Google
ranks me lower because of some standard TYPO3 shortcut pages?
Yeah, there's lots of buzz around, but dup content is fact. Probably a single duplicate page will not harm much, but the misconception of shortcuts MUST lead into duplicate content problems. The more u use them the bigger the problem will grow.
Here is something on this topic (de resource):
http://www.goldmann.de/google-duplicate-content-ist-ein-problem_tipp_198.html
for english resources ask Big G, there will be plenty.
Updated by Daniel Gercke over 18 years ago
Hi,
i´ve build a litte patch on 3.8.1 which should resolve shortcuts directly to the according URL without redirection.
Feel free to try it out. It would be great if someone from content rendering team could take a look at it to make sure that it doesn´t break anything else.
I´m not sure that my patch hit all points were links will be processed.
Actually it will patch two files in tslib (class.tslib_menu.php and class.tslib_content.php).
Updated by Ernesto Baschny over 18 years ago
Daniel:
the patch looks easy, nice work. I haven't tried it yet, thou.
Just some ideas:
Have you tried adding the changes to the t3lib_TStemplate::linkData function, instead of cObj::typoLink and tslib_menu::link? linkData is known as the "mother of all typo3-links", this should always be called if one wants to link to a specific TYPO3-page. So this seems to be the deepest point you could add your changes to so that every extension should be able to work with it.
Or are there any other hidden problems with that?
Updated by Oliver Klee over 18 years ago
Some formatting nits that might improve the code readability (it's your decision if you use them in your patch):
+ if ($page['doktype']==4) {
Use spaces around the == (both times).
+ $tmp_page = $GLOBALS['TSFE']->getPageShortcut($page['shortcut'],$page['shortcut_mode'],$thisUid = $page['uid']);
Use spaces after the commas.
Don't make an assignmend in the function call parameter (side effects decrease readability a lot) - move it on a separate statement. The same for the other function call.
+
+ // Shortcut:
Drop the trailing tabs on the empty line.
The "// Shortcut" comment is aligned differently both times: One time with the brace and one time one tab to the right.
+ if ($tmp_page) { // function getPageShortcut can break if too many shortcut->shortcut. if so fallback to old method
+ $this->menuArr[$key] = $tmp_page;
+ $MP_params = ''; // clear MP parameters since ID was changed.
+ }
The inner indentation is two tabs but should only be one.
Updated by Martin Kutschker over 18 years ago
Daniel, what's different from the standard behaviour? Do try to resolve the link when creating the menu? This won't work for shortcut_mode "random".
Updated by Daniel Gercke over 18 years ago
Thank you for your promt answer Ernesto.
A litte change and the new patch will now affect t3lib/class.t3lib_tstemplate.php. So that resolving is now done by "mother of all typo3-links" :-)
I will post the new patch in a few seconds.
But there is some thing i want to discuss:
Should resolving done before or (maybee and) after $overrideArray is processed?
note added:
sorry but your right. the function getPageShortcut wont give a result if shortcut_mode is 1 or 2. i will try to find the problem.
but no special mode (only direct shortcut) should work with this patch.
if you had any ideas why getPageShortcut didn´t work for shortcut_mode 1&2 please let me know.
Updated by Daniel Gercke over 18 years ago
Hello again,
the problem, that shortcuts in "random subpage" or "first subpage" mode didn´t resolving correct, is gone with patch_shortcut_new.txt on my local testserver.
Here on my local machine all kind of shortcuts will resolved correct to their destination. In "random" Mode - every reload another destination.
My local testsystem ist typo 3.8.1.
Please report if you found a problem - best with example.
Updated by Martin Kutschker over 18 years ago
Daniel, is caching on? I doub't it. The "random" link is fixef once the page is cached.
So I suggest that "random" shortucts are handled with a HTTP redirect. And only "first subpages" are handled like this.
Updated by Daniel Gercke over 18 years ago
Hello,
Martin your right. Caching wasn´t on.
In my opinion there are 2 possible solutions.
Solution 1: Your suggested HTTP redirect.
Solution 2: Don´t cache the affected pages.
I think HTTP redirect is the better solution. But at this moment i don´t have an idea were to hook into page generating to build a patch.
Updated by gregory over 17 years ago
Any news on this isue?
It's still in 4.0.4...
Updated by Martin Kutschker over 17 years ago
Gregory, this won't make it into 4.0 as it's a new feature, but - as there is a feature freeze for 4.1 - it won't make it into 4.1 either. Sorry.
But it seems that only a few think if this issue as urgent. I myself have no problem with the current behaviour.
Updated by Neike Taika-Tessaro about 17 years ago
If a patch has been supplied, is it that hard to integrate? Urgency or not, surely the benefit of this is obvious, even assuming for a moment said benefit only applies select few?
I mean no spite - these are honest questions. Despite being a software developer myself, I'm entirely uninvolved with Typo3 development (extension or otherwise) currently, and as such I don't know what your coding standards are, and so forth -- and I'm not exactly privvy on discussions about this that might have occurred elsewhere.
I've read about - and respect - the feature freeze for 4.1, but are there any plans to integrate this soon? -ish? I'm curious.
Count me as one of the people who would really like to see this, by the way. =)
Updated by Ernesto Baschny about 17 years ago
Neike,
the problem here is the good old problem of backwards compatibility. If we change the current shortcut behaviour from displaying the destination page (behaves like "show content from page") to having it link directly to the destination page might have some consequences that we cannot expect. So at least we (the patch creator) should make it configurable so that old sites behave like they have always done. Imagine hundrets of links that won't work anymore because they are not generated by typolink anymore.
And now testing the proposed patch I think that changing the links to the destination page is not the best way to solve it. I would rather generate a redirect (moved permanten) when we HIT the shortcut page. So that the link still points to the shortcut page (there might be relevat realurl keywords for SEO in it...) but the destination won't show at that place. E.g:
Site structure:
uid=1 Home
uid=2 |-- Page 1
uid=3 | `-- Page 1.1 > shortcut to Page 2 Page 2
uid=4 |-
Currently the link to uid=3 will link to uid=3. When we hit index.php?id=3, the content of page-id=4 is shown.
The patch modifies it so that the link to Page 1.1 will generate a link to uid=4 instead.
My idea would be to keep the link as it is but then change the output of the shortcut destination page (when rendering uid=3):
- shortcut to first, redirect permanent
- shortcut to page, redirect permanent
- shortcut to random subpage, redirect temporary
We might argue if "redirect permanent" is ok here, because the generated links might change in time if the editor change the shortcut somehow.
The problem with this is that the place where getPageShortcut is called might not only be done when rendering it, but maybe from other (extensions?) places that call fetch_the_id() and don't expect a redirect to happen there.
So any patch to do that, have it configurable via TypoScript, default that setting to "true" and have a compatVersion set it to false in case we are upgrading from a lower version?
Updated by Martin Kutschker about 17 years ago
For practical reasons it is clear that random subpages must be done with a redirect. I'm not sure if we really need a config switch for this one. Installation wide as TYPO3_CONF_VARS or page centric with page TS?
For the fixed and first subpages options TYPO3 should generate a 307 response. A config may bee needed, but again I'm not so sure where the benefit of the old behaviour lies.
What should be optional is to resolve the linkes to fixed and first subpages during link time. IMHO it's a nice one to have in a addition to the redirect.
Updated by Cross-Systems about 14 years ago
Any news about that issue ? Still seems to be the case in 4.3.2 at least. I'll try on the 4.4.0
Updated by Chris topher about 14 years ago
I have just split this in several issues, as I think finding solutions to smaller parts of the problem will be easier. For the issue in SimulateStatic please check #23106.
Furthermore I changed the title to something more neutral.
Updated by Michel Tremblay almost 14 years ago
I think this can also be a problem for people trying to get statistics from apache logs. Since the same page can be requested with more than one url, it is difficult to calculate hits for a specific page.
Updated by demonkoryu almost 14 years ago
Hi, this patch also works for pages with doktype other than 4 (shortcut).
Updated by Ernesto Baschny almost 14 years ago
Thanks, demonkoryu, but I think the problem is still the same.
This change of behaviour might have many side-effects which is why I still prefer my solution (no patch available yet, thou) of redirecting to the destination page when the "shortcut page" is reached. This allows the original URLs to continue working and allows more flexibility for example with the "random subpage" stuff.
See my note 14522.
Updated by demonkoryu almost 14 years ago
Hi Ernesto, thanks for your comment.
It could certainly be an optional feature. However, this "just works" with autogenerated links (such as in breadcrumbs), and, as it does not require an additional HTTP request as with redirects, it ensures an overall smoother user experience.
The redirect functionality would be an additional feature that ensures that the URL is valid along the whole trail of partial paths. But with my patch, it needs someone to manually enter the URL to arrive at an "incomplete" path.
I could try to make the "resolve shortcut path" functionality configurable (per TS, in the page properties, or both?), and you could extract the redirect functionality?
Any thoughts?
Updated by Ernesto Baschny almost 14 years ago
Reaching the "old URL" is still required, because it might be linked by other means.
So adding the redirect on the shortcut page is valid even if you never link to it directly anymore in typoLink, mainly also because of backwards compatibility issue and bookmarks / search engines which might have indexed that page already.
This doesn't solve the problem of "random page" shortcut, but I would have to think harder on that.. :)
Updated by Pavel Antonov over 13 years ago
I have created extension that implement HTTP redirect for shortcuts. TER key 'shortcut_redirect'. Also attach extenstion to this ticket. Take a look.
Updated by Thorben Nissen about 13 years ago
- File shortcut_patch.diff shortcut_patch.diff added
- Target version deleted (
0)
Hi there, I want to bring this back to live.
I hit the mentioned problem several times at work. So I thought about patching and found this bug report. There is no need to discuss the duplicate content stuff anymore, but I tried to solve it somehow, so you may have a look at it.
First I changed the menu generation to link directly to target page to avoid the two or more urls for the same page. This is very annoying especially in connection with realurl, cooluri or others. The shortcuts for pages and for external urls are linked directly to the target.
Second I made the tslib_fe to redirect if a page is a shoutcut instead of showing the target page's content (the SEO thing).
Well, take a look and tell me what you think.
I did this using 4.5.3
Regards
Thorben
Updated by Stefan Neufeind about 13 years ago
Current review for that is at: https://review.typo3.org/3398
Updated by Chris topher about 13 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
Updated by Mario Colombo almost 10 years ago
Updated by Lars Zimmermann over 9 years ago
Hi guys,
as Mario said the possibility to choose whether to use 307 or 301 for shortcuts has been removed. Could you pls make this at least configurable again in TYPO3 6.2 LTS ?
From a SEO point of view it's really bad to not have a startingpage with 301 header.
Thank you very much
Lars