Almost the most remote place I know of in Britain lies just west of Nayland, between Sudbury and Harwich.

There are no towpaths on this stretch of the River Stour, no anglers or pedestrians. Instead, there are bulrushes, water lilies and meadows. And, if you paddle around a gentle meander, you will find yourself in a little bit of paradise.

When the sun shines through the clear water on this bend, a little after the A134, it turns endless freshwater mussel shells into a glinting mother of pearl bed.

Above the mussels, the fish are so abundant that you push them away with your paddle. You are 75 miles from London and you are in a spot as beautiful as any in the world.

We don’t know how to use waterways any more. Rivers were once our transport, our water supply, our food source and our sewer. A clause in the Magna Carta even legislated on fish traps, to help to keep our rivers navigable.

But after a year spent canoeing, swimming and surfing along our waterways for my new television series about Britain’s rivers, I have discovered that, in England and Wales at least, they no longer belong to the people. They belong to stockbrokers, to anglers and to farmers.

Private fishing clubs have bought up the banks, controlling the water to the middle — where they meet rival fishing clubs. You cannot pass without permission. Our dislocation from our watery heritage has happened so slowly that no one has really noticed.

It is different in Scotland. The right to roam, that great enlightened law of the ancient Scots, also enshrines the right to paddle. Throughout the country you can go on heroic treks, from the wild rivers of the Highlands — where the Tay cuts a stately path across Rannoch Moor — to the managed flow of the industrial Clyde. But when you reach England, this all changes.

From Hay-on-Wye, on the Welsh borders, down through 40 miles of winding Herefordshire meanders, there is no automatic right of access to the Wye.

Only when you reach Ross-on-Wye can you return your canoe to the water. It is not that Wales is generally much better than England. I think — I hope — it would shock people to learn that of 15,000 miles of Welsh river, only on 300 can the public canoe.

When I started canoeing I had no idea this was the case. One weekend my family and our dog Cadbury — complete with lifejacket — launched ourselves on to a forbidden river.

There was barbed wire across it, and we ended up drifting into some banker’s mill pond. And I thought: how weird. Stranger still are the anglers. To my mind, I am no more of an intrusion on their fishing than a floating log. But they don’t see it that way.

I suppose if mine were just an appeal to maintain a bucolic idyll for my own paddling pleasure, then I would be in danger of slipping into self-interested sentimentality. But rivers are as much a part of our urban history as our natural heritage. Wesley once said that a town that isn’t on a river feeds only itself — and until very recently there was no counterexample to that claim.

Some towns can only be understood when you approach them by river. The meander around Shrewsbury is so pronounced that it might as well be a moat. Canoeing around, under the town’s bridges, you suddenly realise — yes! — that’s why they built it here.

Some of my favourite experiences have been on urban rivers. Often, drifting around the back of a town development, I felt more like a cat burglar than a canoeist. Following the Lee, I paddled into Stanstead Abbotts and suddenly knew what a backwater is — we were floating in these still, quiet waters behind people’s houses.

Some people have untidy gardens, some have a swing seat by the river, some have a garden full of canoes. Then a bit farther on there’s the most exquisitely manicured lawn. It’s illicit, it’s voyeuristic, but it’s wonderful.

Our paddles ended earlier this year, in Constable country. Today we conceive of Constable as a man who painted impossible rural idylls, but one of his most famous pictures is titled Scene on a Navigable River. What he showed is a real river, with barges and carthorses.

Today we need to preserve the idea that rivers are for everyone. I have met anglers, farmers, water companies and industry, and they are all using legislation to fight each other. They say: “This part of the river is for irrigation, this part is for fishing — that part over there is for preservation.” This misses the point. We need to start measuring a river’s intrinsic value, not its monetised worth.

A river is, for all of us, a great spiritual resource. We are healed by rivers: they are wondrous natural corridors through the countryside or the town. That is their value, and if we allow ourselves to be pushed off them, we lose access to a great natural British marvel.

Rivers Journeys with Griff Rhys Jones starts July 26, BBC One